From 5721861564e413bd36e9dfa2049efb133b52c1d0 Mon Sep 17 00:00:00 2001 From: Gabor Szabo Date: Wed, 12 Aug 2026 07:04:08 +0200 Subject: [PATCH 1/2] docs(docs): add a three-track user manual and absorb the user guide (#428) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add docs/manual/ — 23 chapters across an operator, analyst, and integrator track, plus shared configuration, troubleshooting, FAQ, and glossary references. Each chapter carries a Purpose / Intended reader pair, a "What you'll accomplish" section, and a Next pointer. Stated numbers trace to app/core/config.py, a fixed constant, or an authoritative module; runtime figures are explained rather than asserted. Corrects two drifts found against the code while writing: - the model list was 7 types; app/shared/model_taxonomy.py defines 11 - the data-platform table count was 7; there are 10 Six user-guide files become redirect stubs. showcase-manual-demo-guide.md (reviewer QA procedure) and showcase-walkthrough.md (unshipped roadmap) are kept intact with a pointer banner — neither is user-manual content. Every internal link and anchor resolves; no chapter is orphaned. --- docs/manual/README.md | 56 +++++ docs/manual/analyst/backtesting.md | 120 ++++++++++ docs/manual/analyst/champion-selector.md | 152 +++++++++++++ docs/manual/analyst/chat-and-knowledge.md | 147 ++++++++++++ docs/manual/analyst/dashboard-tour.md | 109 +++++++++ docs/manual/analyst/demand-and-planning.md | 106 +++++++++ docs/manual/analyst/forecasting.md | 159 +++++++++++++ docs/manual/configuration.md | 197 ++++++++++++++++ docs/manual/faq.md | 100 +++++++++ docs/manual/glossary.md | 132 +++++++++++ docs/manual/integrator/api-reference.md | 196 ++++++++++++++++ .../integrator/artifacts-and-registry.md | 138 ++++++++++++ .../manual/integrator/ci-and-quality-gates.md | 118 ++++++++++ docs/manual/integrator/code-architecture.md | 125 +++++++++++ docs/manual/integrator/data-model.md | 120 ++++++++++ docs/manual/integrator/extending.md | 135 +++++++++++ docs/manual/operator/concepts.md | 72 ++++++ docs/manual/operator/installation.md | 152 +++++++++++++ docs/manual/operator/operations.md | 119 ++++++++++ docs/manual/operator/quickstart.md | 106 +++++++++ docs/manual/operator/running-the-stack.md | 105 +++++++++ docs/manual/operator/seeding-data.md | 116 ++++++++++ docs/manual/troubleshooting.md | 176 +++++++++++++++ docs/user-guide/advanced-forecasting-guide.md | 181 +-------------- docs/user-guide/agents-and-rag-guide.md | 121 +--------- docs/user-guide/champion-selector-guide.md | 127 +---------- docs/user-guide/dashboard-guide.md | 123 +--------- docs/user-guide/feature-reference.md | 210 +----------------- docs/user-guide/getting-started.md | 103 +-------- docs/user-guide/showcase-manual-demo-guide.md | 5 + docs/user-guide/showcase-walkthrough.md | 5 + 31 files changed, 3007 insertions(+), 824 deletions(-) create mode 100644 docs/manual/README.md create mode 100644 docs/manual/analyst/backtesting.md create mode 100644 docs/manual/analyst/champion-selector.md create mode 100644 docs/manual/analyst/chat-and-knowledge.md create mode 100644 docs/manual/analyst/dashboard-tour.md create mode 100644 docs/manual/analyst/demand-and-planning.md create mode 100644 docs/manual/analyst/forecasting.md create mode 100644 docs/manual/configuration.md create mode 100644 docs/manual/faq.md create mode 100644 docs/manual/glossary.md create mode 100644 docs/manual/integrator/api-reference.md create mode 100644 docs/manual/integrator/artifacts-and-registry.md create mode 100644 docs/manual/integrator/ci-and-quality-gates.md create mode 100644 docs/manual/integrator/code-architecture.md create mode 100644 docs/manual/integrator/data-model.md create mode 100644 docs/manual/integrator/extending.md create mode 100644 docs/manual/operator/concepts.md create mode 100644 docs/manual/operator/installation.md create mode 100644 docs/manual/operator/operations.md create mode 100644 docs/manual/operator/quickstart.md create mode 100644 docs/manual/operator/running-the-stack.md create mode 100644 docs/manual/operator/seeding-data.md create mode 100644 docs/manual/troubleshooting.md diff --git a/docs/manual/README.md b/docs/manual/README.md new file mode 100644 index 00000000..07e1fe84 --- /dev/null +++ b/docs/manual/README.md @@ -0,0 +1,56 @@ +# User manual + +The complete operating, analysis, and integration manual for ForecastLabAI: how to install and run the stack, how to use the dashboard to forecast demand and choose a model, and how to build against the REST API, the data model, and the artifact registry. + +**Purpose:** one navigable place that takes a reader from an empty checkout to a running system with trained models — and explains every number the system reports. +**Intended reader:** anyone running ForecastLabAI for the first time, anyone using the dashboard to make a demand decision, and anyone integrating against the API or extending the code. + +This manual documents behavior; it does not define it. The authoritative contracts live in the code and its generated OpenAPI schema at [`/docs`](http://localhost:8123/docs), and the agent-facing deep dives live in [`docs/_base/`](../_base/). Where this manual and the running system disagree, the system wins — and the disagreement is a bug in this manual. + +## Three tracks + +**Operator track** — you want to *run* ForecastLabAI: stand up Postgres, migrate, seed data, run the pipeline, and keep it healthy. + +1. [What ForecastLabAI is](operator/concepts.md) — the lifecycle, the vocabulary, and what this system is not. +2. [Installation](operator/installation.md) — prerequisites, `.env`, Docker, `uv sync`, migrations. +3. [Quickstart](operator/quickstart.md) — the shortest path to a working system with trained models. +4. [Seeding data](operator/seeding-data.md) — The Forge: scenarios, generation, append, verify, clear. +5. [Running the stack](operator/running-the-stack.md) — backend, frontend, Docker Compose profiles, ports. +6. [Operations](operator/operations.md) — jobs, batches, artifacts, logs, health, and routine upkeep. + +**Analyst track** — you want to *use* ForecastLabAI: explore the data, train and compare models, and decide what to stock. No terminal required. + +1. [Dashboard tour](analyst/dashboard-tour.md) — every page in the web app, grouped as the nav groups them. +2. [Forecasting](analyst/forecasting.md) — the eleven model types, the three families, and the V1/V2 feature frame. +3. [Backtesting](analyst/backtesting.md) — how accuracy is measured, and what each metric does and does not tell you. +4. [Champion selector](analyst/champion-selector.md) — the guided compare → decide → train → promote workflow. +5. [Demand and planning](analyst/demand-and-planning.md) — the Demand Planner and the What-If scenario planner. +6. [Chat and knowledge](analyst/chat-and-knowledge.md) — the two agents, the RAG knowledge base, and the approval gate. + +**Integrator track** — you want to *build on* ForecastLabAI: call its API, read its artifacts, or add to it. + +1. [API reference](integrator/api-reference.md) — the shared conventions, the error envelope, and every endpoint group. +2. [Code architecture](integrator/code-architecture.md) — the vertical-slice layout and the import rules that hold it together. +3. [Data model](integrator/data-model.md) — the retail tables, the registry tables, and how they relate. +4. [Artifacts and the registry](integrator/artifacts-and-registry.md) — run lifecycle, artifact integrity, and aliases. +5. [Extending ForecastLabAI](integrator/extending.md) — adding a model, a slice, or a migration, and what must not change. +6. [CI and quality gates](integrator/ci-and-quality-gates.md) — the five gates, the pipeline, and the release flow. + +**Shared references** — used by all three tracks: + +- [Configuration reference](configuration.md) — every `Settings` field and the environment variables that set them. +- [Troubleshooting](troubleshooting.md) — symptom → cause → fix. +- [FAQ](faq.md) +- [Glossary](glossary.md) — the product vocabulary, used consistently across this manual. + +## Prerequisites, once + +Everything in this manual assumes: Docker and Docker Compose, Python 3.12 with [`uv`](https://docs.astral.sh/uv/), and Node.js 20+ with `pnpm` (via `corepack`) for the dashboard. Everything runs on a single host — there is no cloud account, no managed service, and no multi-tenant deployment anywhere in this system, by design. + +An LLM API key is optional. Forecasting, backtesting, the registry, and the entire dashboard work without one; only the chat agents and OpenAI-backed RAG embeddings require `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`. Embeddings can run fully local through Ollama instead. + +## How this manual reports numbers + +ForecastLabAI ships a **synthetic data generator**, not a real retail dataset. Every accuracy metric you will see — MAE, sMAPE, WAPE, bias, RMSE — is measured against data the system generated from a seed. Those numbers are reproducible, and they are real measurements of model behavior, but they are **not** evidence about real-world retail demand, and this manual never presents them as such. + +Where this manual quotes a number, it is either a configured default (traceable to [`app/core/config.py`](../../app/core/config.py) and repeated in the [configuration reference](configuration.md)) or a fixed constant in the code. Runtime figures — how long a run takes, what WAPE a model achieves, which model wins — depend on your data, your seed, and your hardware, so this manual describes how to read them rather than asserting values it cannot reproduce on your machine. diff --git a/docs/manual/analyst/backtesting.md b/docs/manual/analyst/backtesting.md new file mode 100644 index 00000000..eb56deb0 --- /dev/null +++ b/docs/manual/analyst/backtesting.md @@ -0,0 +1,120 @@ +# Backtesting + +How ForecastLabAI measures accuracy, what each metric means, and what none of them can tell you. + +**Purpose:** read a backtest result correctly, including its limits. +**Intended reader:** analysts comparing models, and anyone about to quote a number. + +## What you'll accomplish + +A defensible answer to "how accurate is this model?" — and the judgement to know when that answer does not transfer. + +## What a backtest is + +A backtest replays history: train on data up to a point, predict forward, compare against what actually happened, then slide the window and repeat. Each train/test split is a **fold**. + +This is not the same as a train/test split on shuffled rows — order matters. Shuffling time-series data lets a model learn from the future, which is the thing everything in this system is built to prevent. See [Forecasting](forecasting.md#leakage-the-guarantee-underneath-all-of-this). + +```bash +curl -X POST http://localhost:8123/backtesting/run -H 'Content-Type: application/json' -d '{...}' +``` + +In the dashboard: **Visualize → Backtest Results**, which can also launch the run in-page. + +## Rolling versus expanding + +**Rolling** — the training window is a fixed width that slides forward. Each fold trains on the same *amount* of data, from a different period. Use it when you believe recent history matters more, or when you want folds that are directly comparable to each other. + +**Expanding** — the training window starts at a minimum and grows. Each fold trains on everything up to its cutoff. Use it when you want to mirror how a model would actually be retrained in production, accumulating history. + +Expanding folds are not comparable to each other in difficulty — later folds have more data — so a trend across folds partly reflects training-set size, not just period difficulty. Keep that in mind before reading a rising accuracy curve as improvement. + +Three settings bound the design: `backtest_max_splits` (20), `backtest_default_min_train_size` (30 days), and `backtest_max_gap` (30). + +## The metrics + +| Metric | What it is | Read it when | +|---|---|---| +| **MAE** | Mean absolute error, in units | You want an answer in units: "off by 4 units/day". | +| **sMAPE** | Symmetric mean absolute percentage error | You want a scale-free number and demand is comfortably above zero. | +| **WAPE** | Total absolute error ÷ total actual demand | **The default.** Scale-free *and* stable at low volume. | +| **Bias** | Signed average error | You care about *direction* of error, which inventory always does. | +| **RMSE** | Root mean squared error | Large misses matter disproportionately. | +| **Stability** | Consistency of error across folds | You care whether the model is reliable, not just good on average. | + +### Why WAPE is the default + +sMAPE is the intuitive scale-free choice, and it misbehaves exactly where retail data lives. As actual demand approaches zero, the percentage error explodes — a single-unit miss on a day with one sale is a 100% error, and a day with zero sales is undefined or degenerate. Intermittent-demand SKUs are common, so a metric that is unstable there is a bad ranking key. + +WAPE divides *total* absolute error by *total* actual demand across the whole window. Low-volume days contribute proportionally to their volume instead of dominating. It stays scale-free, so different SKUs are comparable, without the near-zero pathology. + +WAPE still has a failure case: if total actual demand over a fold is zero, it is undefined. Two seeder presets deliberately tune their noise to avoid that trap — see [Seeding data](../operator/seeding-data.md). + +### Bias has a direction, and the direction matters + +**Positive bias means the model under-forecasts** — it predicted less than actually sold, and following it risks **stockouts**. + +**Negative bias means the model over-forecasts** — it predicted more than sold, and following it risks **overstock** and tied-up cash. + +This is the one metric where the sign carries a business decision, and it is easy to read backwards. Two models with identical MAE and opposite bias fail in opposite, non-interchangeable ways. + +### Stability is a risk measure + +A model averaging 12% WAPE across folds by scoring 11%, 12%, 13% is a different proposition from one scoring 4%, 8%, 24%. The averages match; the second is not something to plan inventory against. Read stability alongside the headline number, not after it. + +## Per-horizon buckets + +Accuracy is not uniform across the horizon. Predicting tomorrow is a different problem from predicting five weeks out, and one aggregate number hides that. + +When the response carries `bucketed_aggregated_metrics`, a **Per-horizon-bucket** card splits error by forecast distance: + +| Bucket | Horizon | +|---|---| +| `h_1_7` | Days 1–7 | +| `h_8_14` | Days 8–14 | +| `h_15_28` | Days 15–28 | +| `h_29_plus` | Days 29+ | + +A metric switcher (MAE / sMAPE / WAPE / Bias / RMSE) sits beside the card title. Empty buckets are dropped. Unknown bucket ids from a newer backend are appended alphabetically rather than discarded — forward compatibility by design. + +This card is what tells you a model is excellent for replenishment but unreliable for planning, or the reverse. If your decision has a horizon, read the bucket for that horizon rather than the aggregate. + +The card renders only when the response carries the data; older jobs will not have it. + +## Baseline versus feature-aware comparison + +When the response carries `baseline_results`, a comparison table renders below the bucket card. + +Every baseline in it runs on the **same folds, with identical splits**, as the main model. That is what makes the comparison meaningful — the models faced the same problem. Lower wins on MAE, sMAPE, WAPE, and RMSE. + +If a feature-aware model cannot beat `seasonal_naive` on identical folds, it has not earned its complexity, its training cost, or its forward-forecast limitation. This table is where that gets settled. + +## Batch sweeps + +To compare many models across many pairs at once, use **Visualize → Batch Runner**. Five presets prefill the matrix: + +| Preset | Loads | +|---|---| +| Quick baseline sweep | All five baselines on V1 | +| Feature-aware comparison | Regression / LightGBM / XGBoost / RandomForest / Prophet-like on V2 with default packs | +| Champion/challenger refresh | Champion plus strongest challenger from the registry | +| Stockout-sensitive products | Regression on V2 with inventory + replenishment + returns packs | +| High-WAPE recovery | Every feature-aware model on V2 with default packs | + +A preset overwrites the matrix; you can hand-edit afterwards. The matrix caps at 24 rows by default. Batch scope and concurrency limits are in [Operations](../operator/operations.md). + +## Reading a result honestly + +**A backtest measures historical fit, not future performance.** It says how a model would have done on data it did not train on, in a period that already happened. That is genuinely informative and it is not a guarantee. + +**The data is synthetic.** Every number here is measured against data the Forge generated from a seed. The measurements are real and reproducible; they are not evidence about real retail demand. A model that wins here won on patterns the generator created. See [What ForecastLabAI is](../operator/concepts.md#the-honesty-caveat-about-data). + +**Metrics measure correlation, not causation.** Neither the accuracy numbers nor the feature-importance panel identify *why* demand moved. + +**Suspect leakage before celebrating.** An unusually good score is more often a feature that saw the future than a breakthrough. `app/features/featuresets/tests/test_leakage.py` is the check. + +**Compare like with like.** Two runs are comparable only if they share a grain, have overlapping data windows, and use the same feature-frame version — the Compare page badges the verdict. See [Champion selector](champion-selector.md). + +## Next + +- [Champion selector](champion-selector.md) — turning a comparison into a decision. diff --git a/docs/manual/analyst/champion-selector.md b/docs/manual/analyst/champion-selector.md new file mode 100644 index 00000000..25d0aa15 --- /dev/null +++ b/docs/manual/analyst/champion-selector.md @@ -0,0 +1,152 @@ +# Champion selector + +The guided workflow that turns "which model is best for this store and product?" into a decision someone signed for. + +**Purpose:** compare candidates, decide, train, forecast, and promote — with every gate explained. +**Intended reader:** analysts choosing a model to put into service. + +## What you'll accomplish + +A ranked comparison at one grain, a trained winner, a business reading of its forecast, and — if you approve it — an audited promotion to a registry alias. + +It lives at **`/visualize/champion`** and is served by the `/model-selection/*` API. + +> **The golden rule of promotion:** the app *recommends* a champion; a human *approves* it; and the decision is **recorded**. Promotion is never automatic. + +## The journey + +``` +Select → Run comparison → Results → Decide / override → Train → Forecast → Interpret → Promote +``` + +## 1 · Select and check availability + +Pick a store, a product, a time period, a horizon (1–90 days), and the candidate models. + +The page first checks **data availability** for the pair and recommends a cross-validation split. A pair with too little history is flagged *unusable* and the comparison is refused with `400` — before any compute is spent. That refusal is a real finding: it means you cannot honestly measure anything at that grain yet, not that the tool is broken. + +## 2 · Run the comparison + +`POST /model-selection/runs` submits an **asynchronous** run, returning `202` with a monitor URL. The page polls to a terminal state. + +Each candidate is backtested with time-series cross-validation on the **same folds**, and results are ranked deterministically. + +Concurrency is bounded by `model_selection_global_max_parallel` (default 4); set it to `1` for sequential execution. + +## 3 · Read the ranking + +**Ranking is by WAPE**, with a fixed tie-break chain: + +``` +WAPE → sMAPE → |bias| → MAE +``` + +The chain is fixed so ranking is reproducible — the same results always produce the same winner, with no hidden tie-breaking. Why WAPE leads is explained in [Backtesting](backtesting.md#why-wape-is-the-default). + +The winner, the runners-up, **and any failed candidates** are all shown. A candidate that failed is information, not noise. + +## 4 · Decide — accept or override + +The recommended winner is pre-selected. Two paths: + +- **Accept** → `POST /model-selection/{id}/train-winner` trains the ranked winner. +- **Override** → `POST /model-selection/{id}/train-selected` trains a different candidate. + +Overriding requires confirming an explicit warning that names the recommended model and the WAPE gap, and lets you record a reason. The override is flagged `is_override=true` and audited. + +Override exists because ranking is not omniscient. A model with marginally worse WAPE but better bias direction, better stability, or fewer feature dependencies can be the right operational choice. The system's job is to make that choice **visible and attributed**, not to prevent it. + +A candidate that *failed its backtest* is still override-trainable — training and backtesting are independent operations. + +## 5 · Forecast + +`POST /model-selection/{id}/predict` generates the horizon forecast. The response carries the **peak** and **low** demand days plus a **decision** block. + +> **Capability limit.** A feature-aware model — `regression`, `prophet_like`, `lightgbm`, `xgboost`, `random_forest` — **cannot auto-forecast here.** It needs a future feature frame, and the system will not invent one. The page shows a blocked state and routes you to the [What-If Planner](demand-and-planning.md). + +This is a real constraint on model choice, not an inconvenience: a model that backtests better but cannot forecast forward without assumptions may be the worse practical pick. Weigh it before promoting. + +## 6 · Interpret + +The **business interpretation** panel restates why the model won, the expected demand over the lead time, and the bias risk: + +> Positive bias means the model **under**-forecasts — stockout risk. Negative bias means it **over**-forecasts — overstock risk. + +The **safety stock** panel shows a deterministic heuristic: + +``` +safety_stock = z(service_level) · σ_daily · √(lead_time_days) +expected_demand = average_demand · lead_time_days +reorder_point = expected_demand + safety_stock +``` + +`σ_daily` is the standard deviation of the daily forecast. `z` comes from a fixed service-level table — 90% → 1.2816, 95% → 1.6449, 97.5% → 1.9600, 99% → 2.3263 — snapping to the nearest level in between. Adjust lead time or service level and recompute. + +> **This is a heuristic, and it is labelled as one.** It models demand variability with a constant lead time. It is not a full inventory optimisation — it ignores supply variability, order costs, and capacity — and it **never** influences model ranking. The decision layer is entirely deterministic; **no LLM is involved.** + +## 7 · Promote + +`POST /model-selection/{id}/promote` registers the trained model as a registry `model_run` (transitioned to SUCCESS with a verified artifact) and points a **registry alias** at it. It records a `promotion_decision` audit: approver, alias, run id, decision, reason, and whether it was an override. + +Four preconditions, each returning `422` with its own message: + +| Requirement | Why | +|---|---| +| Valid alias name matching `^[a-z0-9][a-z0-9\-_]*$` | Aliases are addressable identifiers. | +| `approved_by` present | **Promotion is never anonymous.** | +| `acknowledge_non_recommended=true` for an override | Promoting a non-recommended model must be deliberate. | +| The model is trained first | You cannot promote what does not exist. | + +Re-promoting the same alias name repoints the existing alias — registry upsert semantics. + +**Compare and promote stay separate.** Promote performs no ranking or comparison; it only registers and aliases the already-trained champion. Keeping them apart is what makes the audit trail meaningful: the comparison is evidence, the promotion is a decision. + +### The Promote dialog's three gates + +Promoting from the Control Center opens a confirmation dialog gating on: + +1. **Artifact verification.** The dialog auto-fetches the run's SHA-256 result. A failure renders a red callout and the Promote button **stays disabled — no operator override.** A corrupt or missing artifact is not a judgement call. +2. **Worse-WAPE acknowledgement.** If the candidate's WAPE is higher than the current champion's, a red callout shows the exact deltas and requires an explicit checkbox. +3. **Feature-frame-version mismatch acknowledgement.** If the candidate's `feature_frame_version` differs from the champion's, an amber callout warns that the alias's feature contract will silently change, and a checkbox releases the button. + +The alias defaults to `production`. Cancel preserves nothing — both acknowledgements reset. + +Gate 3 is the subtle one: nothing *fails* when the contract changes. A downstream pipeline feeding the alias keeps working while quietly supplying the wrong columns. That is why it is a deliberate acknowledgement rather than a warning banner. + +## Comparability: when two runs can be compared + +Two runs are comparable for champion/challenger evaluation **if and only if all three hold**: + +1. **Same grain** — same `store_id` and `product_id`. +2. **Overlapping data windows.** +3. **Same `feature_frame_version`** — runs predating the field default to V1. + +The Compare page renders a **Champion compatibility** badge with the verdict, and the metrics-diff table adds a feature-frame-version row when either run declares one. + +Comparing across a grain or a frame version is not a stricter-or-looser judgement call; it is comparing answers to different questions. + +## Stale aliases + +The Control Center flags stale aliases with a reason chip, alongside **Alias V** and **Comparable V** columns showing version drift: + +| Chip | Meaning | +|---|---| +| `newer success run` | A newer successful run exists for this grain. | +| `artifact not verified` | The alias's artifact failed SHA-256 verification. | +| `run not success` | The alias points at a failed or archived run. | +| `V mismatch` | The newest comparable run uses a different `feature_frame_version`. | + +`artifact not verified` is the urgent one — something promoted is no longer what it claimed to be. + +## Anti-patterns + +- **Don't promote without checking bias direction.** Two models with equal WAPE and opposite bias fail in opposite ways. +- **Don't promote a worse run by reflex-ticking the acknowledgement.** The checkbox exists to make you read the deltas. +- **Don't cross a feature-frame boundary without verifying your pipeline supplies the columns the new version demands.** +- **Don't treat the safety-stock number as an inventory plan.** It is a labelled heuristic. +- **Don't read a ranking as a causal claim.** Backtest accuracy is historical fit on synthetic data. + +## Next + +- [Demand and planning](demand-and-planning.md) — forecasting forward with explicit assumptions. +- [Artifacts and the registry](../integrator/artifacts-and-registry.md) — the integrity contract behind the promotion gate. diff --git a/docs/manual/analyst/chat-and-knowledge.md b/docs/manual/analyst/chat-and-knowledge.md new file mode 100644 index 00000000..87a5057f --- /dev/null +++ b/docs/manual/analyst/chat-and-knowledge.md @@ -0,0 +1,147 @@ +# Chat and knowledge + +The two chat agents, the RAG knowledge base they draw on, and the approval gate that keeps them honest. + +**Purpose:** use the conversational layer productively, and understand exactly what it is allowed to do. +**Intended reader:** analysts using `/chat` and `/knowledge`. + +## What you'll accomplish + +Grounded answers with citations, an understanding of when an agent pauses and why, and a clear picture of the boundary around agent actions. + +## Prerequisite + +The agents need an LLM API key — `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` — in `.env`, or a local `ollama:` model configured. **Everything else in ForecastLabAI works without one:** forecasting, backtesting, the registry, the Explorer, and every analytical page. + +## The RAG knowledge base + +RAG — retrieval-augmented generation — answers from a body of indexed documents rather than the language model's general training alone. Here it grounds answers in **project documentation**. + +### How indexing works + +1. A document is split into overlapping **chunks** — markdown by heading, OpenAPI specs by endpoint. +2. Each chunk becomes an **embedding**: a numeric vector capturing its meaning. +3. Chunks and embeddings are stored in PostgreSQL via **pgvector**. + +Indexing is **idempotent**. A document is identified by its path and a content hash, so re-indexing unchanged content does nothing, and changed content replaces the old chunks cleanly. You can safely re-index everything on a schedule. + +Chunking is tunable: `rag_chunk_size` (512 tokens), `rag_chunk_overlap` (50), `rag_min_chunk_size` (100). + +### How retrieval works + +A query is embedded the same way and compared against every stored chunk by **cosine similarity**. Chunks above `rag_similarity_threshold` (default 0.7) are returned — up to `rag_top_k` (default 5) — each with a relevance score and a citation. + +**Retrieval returns evidence, not an answer.** It hands back passages; the agent decides what to do with them. That separation is why answers can carry citations you can check. + +If retrieval returns nothing for an obviously relevant question, the threshold is the usual cause — it is a floor, and different embedding models score differently against it. + +### Embedding providers + +Embeddings come from **OpenAI** or a local **Ollama** server. Ollama keeps document content off external services entirely. + +The active provider, model, and vector dimension are visible and changeable under **Admin → AI models** (`GET` / `PATCH /config/ai`), live with no restart. + +**`rag_embedding_dimension` must match the model's output width** — 1536 for OpenAI `text-embedding-3-small`, 768 for `nomic-embed-text`. The vector column has a fixed width, so a mismatch is a schema error, not a quality problem: changing dimension requires a migration and re-indexing the corpus. + +### Using it + +- **Knowledge page** (`/knowledge`) — browse the corpus, run live semantic searches, and see the system state the agents draw on: seeded data, model runs, deployment aliases. +- **Admin → RAG Sources** — index, list, and delete documents. +- **API** — `POST /rag/index`, `POST /rag/retrieve`, `GET /rag/sources`, `DELETE /rag/sources/{source_id}`. + +An empty corpus means the assistant has nothing to cite. Index relevant documentation first — this manual is a reasonable place to start. + +## The two agents + +Built with PydanticAI: + +- **`rag_assistant`** — answers questions from the knowledge base. +- **`experiment`** — can run forecasting experiments: training, backtesting, registry actions, and proposing scenarios. + +### Talking to one + +Use the **Chat** page (`/chat`) or the API: + +| Endpoint | Purpose | +|---|---| +| `POST /agents/sessions` | Open a session, choosing the agent type. | +| `GET /agents/sessions/{id}` | Session status and message history. | +| `POST /agents/sessions/{id}/chat` | Send a message, get the full response. | +| `POST /agents/sessions/{id}/approve` | Approve or reject a pending tool call. | +| `DELETE /agents/sessions/{id}` | Close the session. | +| `WS /agents/stream` | Token-by-token streaming with tool-call events. | + +A session keeps its history, so the agent remembers earlier turns. + +### Tools, and why they are visible + +Agents call **tools** — typed functions that fetch data or perform actions. The chat UI shows every call and its result. + +That visibility is the point. An answer you can trace through "it called `list_runs`, got these rows, then said this" is checkable in a way a bare paragraph is not. Watching the tool trace is the fastest way to understand how an answer was reached — and to notice when an agent is confidently wrong about something it never looked up. + +## The human-in-the-loop approval gate + +Most tools are read-only and run immediately. Tools that **change state** do not. + +When an agent wants to run a gated tool: + +1. The session enters `awaiting_approval` and emits an `approval_required` event. +2. **Nothing happens.** The agent waits. +3. You approve or reject — in the chat, or via `POST /agents/sessions/{id}/approve`. +4. On approval the tool runs; on rejection it is skipped and the agent continues without it. + +A pending approval expires after `agent_approval_timeout_minutes` (default 60), and the tool does not run. + +The gated set is `agent_require_approval`, which ships as: + +``` +["create_alias", "archive_run", "save_scenario"] +``` + +Those are precisely the mutating tools: pointing an alias at a run, archiving a run, and persisting a scenario plan. **An agent can never silently mutate the registry** — a person is always in the loop for consequential actions. + +This is a configured list, not a hard-coded law, which is the honest framing: removing a name from it *would* let the agent act unattended. That is why widening an agent's mutation surface without adding the tool to this list is forbidden by [AGENTS.md](../../../AGENTS.md), and why `save_scenario` was added to the list when the agent gained the ability to save scenarios. + +If the agent seems to stop mid-task, check for a pending approval before assuming it hung. + +## Session limits + +Bounded so an agent cannot run away: + +| Limit | Setting | Default | +|---|---|---| +| Tool calls per session | `agent_max_tool_calls` | 10 | +| Wall-clock per run | `agent_timeout_seconds` | 120 | +| Response tokens | `agent_max_tokens` | 4096 | +| Session lifetime | `agent_session_ttl_minutes` | 120 | +| Concurrent sessions per user | `agent_max_sessions_per_user` | 5 | + +The **Agent Guide** page (`/guide`) shows these live, with the available tools and example prompts. Because it reads the running configuration, it cannot drift from reality the way a document can. + +An agent that gives up partway through a long task has usually hit the tool-call cap or the timeout. + +## Models and fallback + +`agent_default_model` is the primary; `agent_fallback_model` is used when it fails. Both are `provider:model-name` over `anthropic`, `openai`, `google-gla`, `google-vertex`, or `ollama`. + +If **both** fail, the API returns an `agent-fallback-exhausted` problem response — a distinct error type, so "the LLM provider is down" is diagnosable rather than surfacing as a generic 500. + +Choosing an `ollama:` model runs the agent fully locally with no API key. Swap models live at **Admin → AI models**; identifier validation and its three rejection cases are in the [configuration reference](../configuration.md#the-model-identifier-format). + +## A typical exchange + +You ask a question on `/chat` → the `rag_assistant` calls its retrieval tool → the tool runs a semantic search over the corpus → the agent reads the returned passages → it answers, grounded in real documentation, with citations you can open. + +For experiments, the `experiment` agent can additionally trigger training or backtesting, and propose a scenario — pausing for your approval before anything that writes. + +## Using it well + +- **Index first.** An empty corpus produces confident, ungrounded answers. +- **Read the tool trace,** especially when an answer surprises you. +- **Treat an approval prompt as a decision,** not a dialog to dismiss. It is the same gate that protects the registry. +- **Prefer the deterministic surfaces for decisions.** The Champion selector's ranking and the safety-stock heuristic involve **no LLM at all**. The agent is a way to explore and explain; it is not the thing that decides which model wins. + +## Next + +- [API reference](../integrator/api-reference.md) — calling any of this from code. +- [Configuration reference](../configuration.md) — every agent and RAG setting. diff --git a/docs/manual/analyst/dashboard-tour.md b/docs/manual/analyst/dashboard-tour.md new file mode 100644 index 00000000..3bf2018e --- /dev/null +++ b/docs/manual/analyst/dashboard-tour.md @@ -0,0 +1,109 @@ +# Dashboard tour + +Every page in the web app, grouped the way the navigation groups them. + +**Purpose:** know which page answers which question, so you can stop hunting. +**Intended reader:** analysts and anyone driving a demo. No terminal required. + +## What you'll accomplish + +A map of the dashboard: what each page shows, what it needs in order to show anything, and which chapter goes deeper. + +## Before you start + +The dashboard is at **http://localhost:5173** and reads from the backend at `:8123`. Two conditions must hold or every page looks broken in the same way: + +- The backend is running. If every panel reads "Loading…", it is not — see [Troubleshooting](../troubleshooting.md). +- The database has data. A migrated-but-empty database renders zeros and empty tables, which is correct behavior, not a fault. Seed from **Admin → Data seeding** or run `make demo`. + +The nav groups pages as: Dashboard, Showcase, **Explorer** (menu), **Visualize** (menu), Knowledge, Chat, Agent Guide, Admin. A light/dark toggle sits at the right. + +## Dashboard (`/`) + +The landing page and health check. Headline KPI cards — total revenue, units sold, transactions, average unit price, average basket — over a revenue-over-time chart. + +Use it to confirm at a glance that data is loaded and roughly shaped as you expect. Zeros mean an empty database. + +## Showcase (`/showcase`) + +Runs the **entire end-to-end pipeline live in the browser**: seed → features → train three models → backtest → register the winner → alias → agent check. Each step streams in as a status card that flips to pass, fail, or skip, with a summary banner naming the winning model and its accuracy. + +This is the best page for a guided demo — the same evidence as `make demo`, with no terminal. + +Tick **Re-seed first** if the database is empty or stale. Only one pipeline may run at a time. See [Quickstart](../operator/quickstart.md). + +## Explorer + +Read-only pages for browsing data and model history. Every table supports pagination, filtering, search, and server-side sorting, exports to CSV, has column-visibility toggles, and encodes filter/sort/page state **in the URL** — so a view you are looking at is a link you can send someone. + +Detail pages are reached by **clicking a table row**; they are not in the nav. + +- **Sales** (`/explorer/sales`) — daily sales records, with date-scoped KPIs, revenue bar/line charts, and cross-filtering. +- **Stores** (`/explorer/stores`) — the store list. A row opens a store profile with date-scoped KPIs, a revenue-over-time chart, and a top-products drilldown. +- **Products** (`/explorer/products`) — the product list. A row opens a profile with KPIs, revenue and lifecycle-demand curves, and a top-stores drilldown. +- **Model Runs** (`/explorer/runs`) — every trained model in the registry, with a **Family** badge distinguishing baseline, tree, and additive at a glance. The detail page shows configuration, metrics, runtime info, cross-links to the store and product, an artifact-integrity check, and — for non-baseline runs — the canonical feature columns and a feature-importance panel. +- **Jobs** (`/explorer/jobs`) — submitted train, predict, and backtest jobs. The detail page shows parameters, result JSON, error details, the linked run, a cancel action, and live status polling. + +**Comparing two runs.** From the runs list you can open a side-by-side comparison. It carries a **Champion compatibility** badge — the verdict on whether the two runs are legitimately comparable — and a metrics-diff table including a feature-frame-version row. Comparability is not cosmetic; see [Champion selector](champion-selector.md). + +## Visualize + +The analytical, chart-heavy pages. + +- **Demand Planner** (`/visualize/demand`) — every completed forecast rolled into a multi-SKU table: tomorrow, next-week, and next-month demand plus the inventory required to cover it. Includes a lead-time selector and a single-SKU drill-in. Answers "how much will this sell, and do I have enough?" See [Demand and planning](demand-and-planning.md). +- **Forecast** (`/visualize/forecast`) — a model's horizon predictions, with an optional prediction-interval band. The top of the page hosts a **Train a new model** card: family picker, model select, feature-frame V1/V2 select, and feature-pack toggles. See [Forecasting](forecasting.md). +- **Backtest Results** (`/visualize/backtest`) — fold charts and accuracy metrics, with a per-horizon-bucket card and a baseline-versus-feature-aware comparison table when the response carries them. See [Backtesting](backtesting.md). +- **Champion** (`/visualize/champion`) — the guided compare → decide → train → forecast → promote workflow. See [Champion selector](champion-selector.md). +- **What-If Planner** (`/visualize/planner`) — apply price, promotion, holiday, inventory, and lifecycle assumptions to an existing forecast and see the baseline-versus-scenario impact. The impact card carries a **method badge** telling you whether the result came from a real re-forecast or a heuristic adjustment. See [Demand and planning](demand-and-planning.md). +- **Batch Runner** (`/visualize/batch`) — run a matrix of jobs, with five sweep presets and a model × V1/V2 matrix picker. + +Both the Forecast and Backtest pages run jobs **in-page**, export CSV, and cross-link back to runs and jobs. + +## Knowledge (`/knowledge`) + +The RAG knowledge base: the indexed corpus, a live semantic search box, and the current system state the agents draw on — seeded data, model runs, deployment aliases. + +Type a question to retrieve the most relevant documentation passages with similarity scores. An empty corpus shows an empty state until documents are indexed via **Admin → RAG Sources**. See [Chat and knowledge](chat-and-knowledge.md). + +## Chat (`/chat`) + +The AI agent chat. Ask in natural language; the answer streams token by token and every tool the agent calls is displayed with its result. + +Some actions **pause for your approval** before running — that is the human-in-the-loop gate working, not a hang. See [Chat and knowledge](chat-and-knowledge.md). + +Requires an LLM API key. Everything else in the dashboard works without one. + +## Agent Guide (`/guide`) + +An in-product reference for the two agents: the tools they can call, the approval gate, the live session limits, and copy-paste example prompts. Worth opening before your first chat session — the limits shown are the live configured values, not documentation that can drift. + +## Admin (`/admin`) + +Operational controls, in tabs: + +- **Data seeding** — generate synthetic data from a scenario, append more, verify integrity, or clear. See [Seeding data](../operator/seeding-data.md). +- **RAG Sources** — list, index, and delete knowledge documents. +- **Aliases** — manage registry aliases, including promoting a run to `production`. +- **AI models** — swap the agent LLM (including fully local Ollama), the embedding model, and provider API keys **live, with no restart**, with per-provider health indicators. + +The Promote action here opens a confirmation dialog with three gates — artifact verification, a worse-WAPE acknowledgement, and a feature-frame-mismatch acknowledgement. Only the first has no override. See [Champion selector](champion-selector.md). + +## Which page answers which question + +| Question | Page | +|---|---| +| Is the system loaded and healthy? | Dashboard (`/`) | +| Can I see the whole system work? | Showcase | +| What does the underlying data look like? | Explorer → Sales / Stores / Products | +| What models have been trained? | Explorer → Model Runs | +| Why did that job fail? | Explorer → Jobs | +| How much will this SKU sell? | Visualize → Demand Planner | +| How accurate is this model? | Visualize → Backtest Results | +| Which model should I use? | Visualize → Champion | +| What if we ran a promotion? | Visualize → What-If Planner | +| What does the documentation say? | Knowledge, or Chat | +| How do I change the agent model? | Admin → AI models | + +## Next + +- [Forecasting](forecasting.md) — the model families and the feature frame. diff --git a/docs/manual/analyst/demand-and-planning.md b/docs/manual/analyst/demand-and-planning.md new file mode 100644 index 00000000..520cafc6 --- /dev/null +++ b/docs/manual/analyst/demand-and-planning.md @@ -0,0 +1,106 @@ +# Demand and planning + +Turning forecasts into stocking decisions, and asking what would happen if you changed something. + +**Purpose:** use the Demand Planner to see what you need, and the What-If Planner to test a decision before making it. +**Intended reader:** analysts planning inventory or evaluating a promotion. + +## What you'll accomplish + +A multi-SKU view of upcoming demand with inventory requirements, and a saved, comparable scenario showing the impact of price, promotion, holiday, inventory, or lifecycle assumptions. + +## Demand Planner (`/visualize/demand`) + +Rolls **every completed `predict` job** into one table: for each SKU, demand for tomorrow, next week, and next month, plus the inventory required to cover it. + +Controls: a **lead-time selector** that reshapes the requirement columns, and a single-SKU **drill-in** for one product's detail. + +**It shows what has been forecast, not what could be.** With no completed prediction jobs the table is empty — that is the page working correctly on an empty input. Train a model and run a forecast first ([Forecasting](forecasting.md)). + +Because it aggregates completed jobs rather than forecasting on demand, the view is only as fresh as your most recent prediction run. A SKU whose forecast is three weeks old will say so; it will not silently re-forecast. + +## What-If Planner (`/visualize/planner`) + +Takes an existing forecast, applies assumptions, and shows baseline-versus-scenario demand and revenue impact. + +### The five assumption types + +| Assumption | What you state | +|---|---| +| **Price** | A different price level. | +| **Promotion** | A promotion of kind `pct_off`, `bogo`, `bundle`, or `markdown`. | +| **Holiday** | A holiday effect on the window. | +| **Inventory** | An on-hand position, which drives a coverage verdict. | +| **Lifecycle** | A stage: `launch`, `growth`, `maturity`, or `decline`. | + +Inventory assumptions produce a **coverage verdict** — `covered`, `at_risk`, `stockout`, or `unknown`. `unknown` is a real answer, not a failure: it means the inputs do not support a verdict. + +### The method badge — read this first + +Every impact card carries a **method badge**, and it changes what the result means: + +| Badge | Method | What actually happened | +|---|---|---| +| **model-driven re-forecast** | `model_exogenous` | A regression baseline genuinely **re-forecast** through your assumptions. | +| **heuristic adjustment** | `heuristic` | A deterministic post-forecast multiplier was applied. | + +A `model_exogenous` result is a model's answer to "given these inputs, what happens?". A `heuristic` result is arithmetic applied *after* forecasting — transparent and reproducible, but it did not consult the model. + +Both are legitimate; they answer with different authority. The badge exists so you never have to guess which you are looking at, and it is why this page can afford to always return an answer instead of refusing. + +### Why this page exists + +Recall the capability limit from [Forecasting](forecasting.md#the-capability-limit-worth-knowing-early): a feature-aware model cannot auto-forecast forward, because it needs future feature values that do not exist yet. + +The What-If Planner is the resolution. Rather than fabricating those values, it makes you **state them** — and then labels which mechanism produced the result. The Champion selector's blocked forecast state routes here for exactly this reason. Assumptions you declared beat assumptions the system invented on your behalf. + +### Saving, tagging, and comparing + +Scenarios are first-class objects, not throwaway calculations: + +| Endpoint | Purpose | +|---|---| +| `POST /scenarios/simulate` | Run a simulation. | +| `POST /scenarios` | Save a named plan. | +| `GET /scenarios` | List saved plans. | +| `GET /scenarios/{scenario_id}` | Fetch one. | +| `DELETE /scenarios/{scenario_id}` | Delete one. | +| `POST /scenarios/compare` | Rank 2–5 saved plans side by side. | + +Save, tag, reload, clone, and delete named plans; then rank **2 to 5** of them in a multi-scenario comparison, ordered by `revenue_delta` or `units_delta`. + +Comparison is where the page earns its keep. A single scenario tells you an assumption's effect; ranking several tells you which lever is worth pulling. + +Each plan records its `source` — `user` or `agent` — so a scenario the experiment agent proposed is distinguishable from one you built. + +### The agent can propose a scenario + +The experiment agent can propose a scenario and, **behind the human-in-the-loop approval gate**, save it. `save_scenario` is in `agent_require_approval` alongside `create_alias` and `archive_run`, so the agent pauses and waits for you before persisting anything. + +See [Chat and knowledge](chat-and-knowledge.md). + +## Safety stock, once + +The inventory arithmetic surfaced in the Champion selector applies here too: + +``` +safety_stock = z(service_level) · σ_daily · √(lead_time_days) +expected_demand = average_demand · lead_time_days +reorder_point = expected_demand + safety_stock +``` + +It is a **labelled deterministic heuristic** over demand variability with a constant lead time — not a full inventory optimisation, and it never influences model ranking. Full detail in [Champion selector](champion-selector.md#6--interpret). + +## Reading a planning result honestly + +**A scenario is a conditional, not a prediction.** "If we run 20% off, demand rises by X" holds only insofar as the assumption is right and the mechanism behind the badge is trustworthy for your case. + +**Check the badge before quoting a number.** A heuristic adjustment is arithmetic on top of a forecast; do not present it as the model's opinion. + +**The underlying data is synthetic.** Elasticity the Forge generated is elasticity the Forge generated. See [What ForecastLabAI is](../operator/concepts.md#the-honesty-caveat-about-data). + +**Lead time is an input, not a forecast.** Nothing here predicts supplier behavior. + +## Next + +- [Chat and knowledge](chat-and-knowledge.md) — the agents, including the one that proposes scenarios. diff --git a/docs/manual/analyst/forecasting.md b/docs/manual/analyst/forecasting.md new file mode 100644 index 00000000..486d5818 --- /dev/null +++ b/docs/manual/analyst/forecasting.md @@ -0,0 +1,159 @@ +# Forecasting + +The eleven model types, the three families they group into, and the feature-frame contract that decides what a model is allowed to see. + +**Purpose:** pick a model deliberately and know what it can and cannot do. +**Intended reader:** analysts training models from the dashboard or the API. + +## What you'll accomplish + +A trained model at a chosen grain, and the vocabulary to explain why you picked it. + +## The grain + +Every forecast in ForecastLabAI is made for one **(store, product) pair**, one day at a time, out to a horizon. The horizon defaults to 14 days (`forecast_default_horizon`) and is capped at 90 (`forecast_max_horizon`). + +Grain matters beyond training: two runs are only comparable if they share one. + +## The three families + +**Family is a property of the model code, not a label you choose.** It is computed from the model type by `app/shared/model_taxonomy.py` and never stored in the database — it appears on API responses as a computed field and drives the Family badge in the dashboard. + +| Family | Model types | Where it shines | +|---|---|---| +| **Baseline** | `naive`, `seasonal_naive`, `moving_average`, `weighted_moving_average`, `seasonal_average` | Sanity checks, target-only history, very short windows. | +| **Tree** | `regression` (HistGradientBoostingRegressor), `lightgbm`, `xgboost`, `random_forest` | Mid-to-long horizons with rich feature signal. | +| **Additive** | `prophet_like` (Ridge additive), `trend_regression_baseline` | Strong yearly seasonality; interpretable coefficients. | + +Eleven model types in total. Note that `trend_regression_baseline` classifies as **additive**, not baseline, despite its name — it fits a trend rather than repeating history. + +An unknown model type classifies as `baseline` and logs a warning rather than raising, so a model added before the taxonomy map is updated degrades gracefully instead of breaking the dashboard. + +### Availability + +Three model types are opt-in and absent from the picker until enabled: + +| Model | Needs | +|---|---| +| `lightgbm` | `uv sync --extra ml-lightgbm` **and** `FORECAST_ENABLE_LIGHTGBM=true` | +| `xgboost` | `uv sync --extra ml-xgboost` **and** `FORECAST_ENABLE_XGBOOST=true` | +| `random_forest` | `FORECAST_ENABLE_RANDOM_FOREST=true` only — pure scikit-learn | + +The flags are **permission gates, not installation checks**. Setting one without the library succeeds at startup and fails later at fit or unpickle time. All three need a backend restart. + +### Why baselines are first-class + +Five baselines ship as real, rankable models. "Predict the same weekday last week" is a genuinely strong forecaster for seasonal retail demand, and a gradient-boosted model that cannot beat it has earned no complexity budget. + +A baseline winning a comparison is a **legitimate result**, not a misconfiguration. It happens regularly on short histories and synthetic data — and you would never learn it without the baseline in the race. + +## The feature frame: V1 versus V2 + +The feature frame is the versioned contract for what a model sees. + +**V1 — target-only.** Lags plus same-day-of-week means, derived from the sales history alone. Every model in every family can train on V1. **It is the backend default.** + +**V2 — feature-aware.** The richer contract, adding eleven optional feature packs. Available to **tree and additive families only** — baselines ignore features, so the UI disables the combination with a tooltip rather than letting you pick a no-op. + +Two rules the backend enforces: + +- The UI sends `feature_frame_version=2` only when you explicitly pick V2. +- A **V1 request carrying `feature_groups` is rejected with `422`.** Packs are a V2 concept; silently ignoring them would hide a mistake. + +## The eleven feature packs + +Each pack is a named subset of V2 columns you toggle independently. + +| Pack | What it carries | On by default | +|---|---|---| +| `target_history` | Lag features and same-day-of-week means | ✅ | +| `calendar` | Day-of-week, month, sin/cos calendar signals | ✅ | +| `rolling` | Rolling means over multiple windows | ✅ | +| `trend` | 30-day and 90-day trend | ✅ | +| `price_promo` | Price level and promotion indicators | ✅ | +| `lifecycle` | Product lifecycle stage | ✅ | +| `inventory` | On-hand stock and stockout flags | — | +| `replenishment` | Inbound stock cadence | — | +| `returns` | Return intensity | — | +| `exogenous_weather` | Weather signals (when seeded) | — | +| `exogenous_macro` | Macro signals (when seeded) | — | + +The six defaults are what the backend uses when `feature_groups` is omitted. The five off-by-default packs read sidecar tables that a smaller seeded database may not populate meaningfully — enabling one whose signal was never seeded contributes nothing. If you want to study inventory-aware forecasting, seed `stockout_heavy` first. See [Seeding data](../operator/seeding-data.md). + +In the dashboard, **Use defaults** loads the six; **Clear** empties the selection, which forwards no `feature_groups` at all and is therefore treated by the server as the default set — clearing does not mean "no features". + +### Disabled means absent, not blank + +Disabling a pack **omits its columns entirely** — it does not fill them with NaN placeholders. That distinction matters when you read a feature-importance panel: a column that is not there cannot rank. + +A NaN *inside an enabled* pack means something different: "the source data is unknown for this day." The tree models handle NaN natively rather than requiring imputation, so a partially-populated signal is usable rather than fatal. + +### Safety classes + +A pack may carry a safety chip when the server returns a `feature_safety_classes` map: `Safe`, `Conditionally safe`, or `Requires supplied data`. + +**`Requires supplied data` is the one to act on.** It means the pack reads a column your production pipeline must keep populated — inventory or replenishment, typically. Promote a run using such a pack only if you can guarantee that column keeps arriving. A model silently starved of a feature it was trained on does not fail loudly; it just gets worse. + +## Leakage: the guarantee underneath all of this + +A feature that sees the future makes a model look excellent and be worthless. + +Every feature is built so this cannot happen structurally — `shift(lag)` and `shift(1).rolling()` patterns with entity-aware grouping, computed only up to a cutoff date. The guarantee is locked by `app/features/featuresets/tests/test_leakage.py`, which the repository treats as the specification: weakening it is explicitly forbidden. + +Practically: **if a metric looks too good, suspect leakage before celebrating**, and run that test. It is the fastest way to tell a real result from an artifact. + +Three settings bound feature cost — `feature_max_lookback_days` (1095), `feature_max_lag` (365), `feature_max_window` (90). These are budget ceilings, not safety controls; safety comes from the code. + +## Training a model + +### From the dashboard + +**Visualize → Forecast**, in the *Train a new model* card: pick a family, then a model type (the list filters to the family), then the feature frame, then — for V2 — the packs. Submit, and the page tracks the job. + +### From the API + +```bash +# Compute features up to a cutoff +curl -X POST http://localhost:8123/featuresets/compute -H 'Content-Type: application/json' -d '{...}' + +# Train +curl -X POST http://localhost:8123/forecasting/train -H 'Content-Type: application/json' -d '{...}' + +# Predict +curl -X POST http://localhost:8123/forecasting/predict -H 'Content-Type: application/json' -d '{...}' +``` + +`POST /featuresets/preview` returns sample rows without committing, which is the quickest way to see what a pack actually contributes. + +Training runs as a **job** — see [Operations](../operator/operations.md). + +## The capability limit worth knowing early + +**A feature-aware model cannot auto-forecast forward.** To predict day *N+7*, a tree or additive model needs the feature values for that day — future prices, future promotions, future inventory. Those are assumptions, not facts. + +Rather than fabricate them, the system blocks the auto-forecast and routes you to the **What-If Planner**, where you state the assumptions explicitly and see them labelled. Baselines are unaffected: they only need history. + +This is why a baseline can be the *practical* choice even when a feature-aware model backtests better. See [Demand and planning](demand-and-planning.md). + +## Feature importance, and how to read it + +For a non-baseline run, the run detail page shows the canonical feature columns and an importance panel. + +- **Tree family** — non-negative bars from the booster's native `feature_importances_`. The exact meaning varies by library (LightGBM's `split`, XGBoost's `weight`); the panel labels which. +- **Additive `prophet_like`** — signed Ridge coefficients: positive renders green with an up arrow, negative red with a down arrow. The sign is preserved because direction is the interpretable part. + +> **Correlation, not causation.** Importance reflects how much a feature reduced the model's *training* error. It is not evidence about real-world demand drivers, and two products with similar importance profiles need not share a business cause. + +When the panel is unavailable, the status code says why: **400** the run is a baseline (nothing is wrong), **404** the run or job is not in the registry, **422** no artifact yet, artifact deleted, a missing `ml-*` extra at unpickle time, or an estimator that does not expose importances. Note `regression` uses scikit-learn's `HistGradientBoostingRegressor`, which **does not** expose `feature_importances_` — a 422 there is permanent. + +## Choosing a model + +1. **Always include baselines.** They set the bar. +2. **Reach for V2 only with signal to feed it.** V2 on a dataset with no promotions or inventory dynamics adds columns, not information. +3. **Match the family to the question.** Strong yearly seasonality with a need to explain coefficients → additive. Rich features and mid-to-long horizons → tree. Short history → baseline. +4. **Mind the forward-forecast limit.** If you need to forecast forward without stating assumptions, a baseline is the model that can. +5. **Let the backtest decide.** Intuition about which model *should* win is exactly what backtesting exists to check — [Backtesting](backtesting.md). + +## Next + +- [Backtesting](backtesting.md) — measuring whether the model you picked is any good. diff --git a/docs/manual/configuration.md b/docs/manual/configuration.md new file mode 100644 index 00000000..c083e0fa --- /dev/null +++ b/docs/manual/configuration.md @@ -0,0 +1,197 @@ +# Configuration reference + +Every setting ForecastLabAI reads, what it controls, and how to change it. + +**Purpose:** one complete, verifiable table of the configuration surface. +**Intended reader:** operators tuning a deployment and integrators who need to know which knobs exist. + +## How configuration works + +All settings live in one Pydantic Settings class, [`Settings` in `app/core/config.py`](../../app/core/config.py). Each field maps to an environment variable of the same name in **upper case** — `forecast_max_horizon` is set by `FORECAST_MAX_HORIZON`. Values are read from the process environment and from a `.env` file in the repository root; unknown variables are ignored (`extra="ignore"`), so a typo in a variable name fails silently as a *default*, not as an error. + +Application code reads settings through `get_settings()`, which is `@lru_cache`d — a singleton for the process lifetime. Feature code must never touch `os.environ` directly; that rule is in [AGENTS.md](../../AGENTS.md) and is what makes this table complete. + +Three consequences worth knowing: + +- **`.env.example` is a starting point, not the full surface.** It ships the variables most deployments change. Many fields below have no line in it and are configured only by adding one. +- **Most changes need a restart**, because the settings object is cached at first read. +- **Except the AI-model settings**, which are the deliberate exception — see [Runtime-editable settings](#runtime-editable-settings-no-restart) below. + +## Application + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `app_name` | `APP_NAME` | `ForecastLabAI` | Service name in logs and startup events. | +| `app_env` | `APP_ENV` | `development` | One of `development`, `testing`, `staging`, `production`. Drives the `is_development` / `is_testing` / `is_production` properties. | +| `debug` | `DEBUG` | `false` | Debug flag surfaced at startup. `.env.example` ships `true`. | +| `log_level` | `LOG_LEVEL` | `INFO` | One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. | +| `log_format` | `LOG_FORMAT` | `json` | `json` for structured output, `console` for human-readable local logs. | + +## Database and API + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `database_url` | `DATABASE_URL` | `postgresql+asyncpg://forecastlab:forecastlab@localhost:5433/forecastlab` | Async SQLAlchemy connection string. The host-mode default targets the Compose Postgres on host port **5433**. | +| `api_host` | `API_HOST` | `0.0.0.0` | Bind address. The default listens on all interfaces. | +| `api_port` | `API_PORT` | `8123` | Backend port. | + +Running the stack in containers changes the database host: the backend container's `environment:` block sets `DATABASE_URL` to `…@postgres:5432/…` (in-cluster DNS, container port), overriding whatever `.env` holds. This is why the `.env` value stays the *host-mode* default — see [Running the stack](operator/running-the-stack.md). + +The dashboard reads one variable of its own, `VITE_API_BASE_URL` (default `http://localhost:8123`), from `frontend/.env`. It is a Vite build-time variable, not part of `Settings`. + +## Ingest and feature engineering + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `ingest_batch_size` | `INGEST_BATCH_SIZE` | `1000` | Rows per batch in `POST /ingest/sales-daily`. | +| `ingest_timeout_seconds` | `INGEST_TIMEOUT_SECONDS` | `60` | Ingest request timeout. | +| `feature_max_lookback_days` | `FEATURE_MAX_LOOKBACK_DAYS` | `1095` | Ceiling on history a feature computation may read — three years. | +| `feature_max_lag` | `FEATURE_MAX_LAG` | `365` | Largest permitted lag feature, in days. | +| `feature_max_window` | `FEATURE_MAX_WINDOW` | `90` | Largest permitted rolling window, in days. | + +These three feature ceilings bound cost, not correctness. Leakage safety is enforced by the feature code and locked by `app/features/featuresets/tests/test_leakage.py` — see [Forecasting](analyst/forecasting.md). + +## Forecasting + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `forecast_random_seed` | `FORECAST_RANDOM_SEED` | `42` | Seed for model training. Fixing it is what makes a run reproducible. | +| `forecast_default_horizon` | `FORECAST_DEFAULT_HORIZON` | `14` | Default forecast horizon in days. | +| `forecast_max_horizon` | `FORECAST_MAX_HORIZON` | `90` | Maximum accepted horizon; a longer request is rejected. | +| `forecast_model_artifacts_dir` | `FORECAST_MODEL_ARTIFACTS_DIR` | `./artifacts/models` | Where fitted model artifacts are written. | +| `forecast_enable_lightgbm` | `FORECAST_ENABLE_LIGHTGBM` | `false` | Enables the `lightgbm` model type. Also needs the `ml-lightgbm` extra installed. | +| `forecast_enable_xgboost` | `FORECAST_ENABLE_XGBOOST` | `false` | Enables the `xgboost` model type. Also needs the `ml-xgboost` extra installed. | +| `forecast_enable_random_forest` | `FORECAST_ENABLE_RANDOM_FOREST` | `false` | Enables the `random_forest` model type. Pure scikit-learn — no extra dependency needed. | + +The three `forecast_enable_*` flags are **opt-in gates, not installation checks**. Setting a flag to `true` without installing the matching extra will fail when the model is actually trained or unpickled, not at startup. `random_forest` is the exception with no extra to install. + +## Backtesting + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `backtest_max_splits` | `BACKTEST_MAX_SPLITS` | `20` | Maximum cross-validation folds per backtest. | +| `backtest_default_min_train_size` | `BACKTEST_DEFAULT_MIN_TRAIN_SIZE` | `30` | Minimum training rows before the first fold, in days. | +| `backtest_max_gap` | `BACKTEST_MAX_GAP` | `30` | Maximum permitted gap between train and test windows. | +| `backtest_results_dir` | `BACKTEST_RESULTS_DIR` | `./artifacts/backtests` | Where backtest result files are written. | + +## Registry and artifacts + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `registry_artifact_root` | `REGISTRY_ARTIFACT_ROOT` | `./artifacts/registry` | Root directory for registry-tracked artifacts. | +| `registry_duplicate_policy` | `REGISTRY_DUPLICATE_POLICY` | `detect` | One of `allow`, `deny`, `detect` — how a duplicate run registration is handled. | +| `showcase_export_root` | `SHOWCASE_EXPORT_ROOT` | `./artifacts/showcase` | Root for workspace export bundles (manifest plus checksums). | + +## Analytics and jobs + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `analytics_max_rows` | `ANALYTICS_MAX_ROWS` | `10000` | Row ceiling on an analytics response. | +| `analytics_max_date_range_days` | `ANALYTICS_MAX_DATE_RANGE_DAYS` | `730` | Largest queryable date range — two years. | +| `jobs_retention_days` | `JOBS_RETENTION_DAYS` | `30` | How long job records are kept. | + +## Batch runner + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `batch_max_scope_expansion` | `BATCH_MAX_SCOPE_EXPANSION` | `1000` | Cap on expanded scope (pairs × model configs). A batch that would expand past this is rejected rather than queued. | +| `batch_global_max_parallel` | `BATCH_GLOBAL_MAX_PARALLEL` | `4` | Host-wide ceiling on concurrent batch items across *all* active batches. Effective per-batch parallelism is `min(batch_job.max_parallel, this)`. | +| `batch_cancel_drain_timeout_seconds` | `BATCH_CANCEL_DRAIN_TIMEOUT_SECONDS` | `30` | How long `DELETE /batch/{batch_id}` waits for in-flight children before returning a 504. | + +The default of `4` is sized for the Compose Postgres pool (`pool_size=5`, `max_overflow=10`). Raising it without raising the pool will surface as connection-pool exhaustion under load. Both parallelism settings require a backend restart. + +## Champion selector (model selection) + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `model_selection_global_max_parallel` | `MODEL_SELECTION_GLOBAL_MAX_PARALLEL` | `4` | Host-wide ceiling on concurrent candidate backtests. Set to `1` for sequential execution. | +| `model_selection_cancel_drain_timeout_seconds` | `MODEL_SELECTION_CANCEL_DRAIN_TIMEOUT_SECONDS` | `30` | How long `DELETE /model-selection/{id}` waits for in-flight candidates before returning a 504. | + +Both drain timeouts exist because an in-flight scikit-learn or LightGBM fit **cannot be cancelled mid-call**. The timeout bounds how long the API will wait for a fit to finish on its own before giving up and reporting a 504. + +## RAG: embeddings, chunking, retrieval, index + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `rag_embedding_provider` | `RAG_EMBEDDING_PROVIDER` | `openai` | `openai` or `ollama`. Choosing `ollama` keeps document content off external services. | +| `openai_api_key` | `OPENAI_API_KEY` | *(empty)* | OpenAI credential, for embeddings and/or the agent. | +| `rag_embedding_model` | `RAG_EMBEDDING_MODEL` | `text-embedding-3-small` | Embedding model name for the OpenAI provider. | +| `rag_embedding_dimension` | `RAG_EMBEDDING_DIMENSION` | `1536` | Vector width. **Must match the chosen model.** | +| `rag_embedding_batch_size` | `RAG_EMBEDDING_BATCH_SIZE` | `100` | Chunks embedded per API call. | +| `ollama_base_url` | `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint. In GPU Compose mode the backend injects `http://ollama:11434`. | +| `ollama_embedding_model` | `OLLAMA_EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model for the Ollama provider. | +| `rag_chunk_size` | `RAG_CHUNK_SIZE` | `512` | Target chunk size, in tokens. | +| `rag_chunk_overlap` | `RAG_CHUNK_OVERLAP` | `50` | Token overlap between adjacent chunks. | +| `rag_min_chunk_size` | `RAG_MIN_CHUNK_SIZE` | `100` | Minimum tokens for a chunk to be kept. | +| `rag_top_k` | `RAG_TOP_K` | `5` | Passages returned per retrieval. | +| `rag_similarity_threshold` | `RAG_SIMILARITY_THRESHOLD` | `0.7` | Cosine-similarity floor. Passages below it are not returned. | +| `rag_max_context_tokens` | `RAG_MAX_CONTEXT_TOKENS` | `4000` | Ceiling on retrieved context handed to an agent. | +| `rag_index_type` | `RAG_INDEX_TYPE` | `hnsw` | pgvector index type: `hnsw` or `ivfflat`. | +| `rag_hnsw_m` | `RAG_HNSW_M` | `16` | HNSW graph degree. | +| `rag_hnsw_ef_construction` | `RAG_HNSW_EF_CONSTRUCTION` | `64` | HNSW build-time search width. | + +**`rag_embedding_dimension` is the setting that most often breaks RAG.** It must equal the width the embedding model actually emits — OpenAI `text-embedding-3-small` is 1536, `nomic-embed-text` is 768. A mismatch is a schema-level failure, not a quality problem: the stored vector column has a fixed width. Changing dimension means a migration, not just a settings edit. See [Troubleshooting](troubleshooting.md). + +## Agent LLM and execution + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `agent_default_model` | `AGENT_DEFAULT_MODEL` | `anthropic:claude-sonnet-4-5` | Primary agent model, as `provider:model-name`. | +| `agent_fallback_model` | `AGENT_FALLBACK_MODEL` | `openai:gpt-4o` | Model used when the primary fails. | +| `agent_temperature` | `AGENT_TEMPERATURE` | `0.1` | Sampling temperature. | +| `agent_max_tokens` | `AGENT_MAX_TOKENS` | `4096` | Response token ceiling. | +| `anthropic_api_key` | `ANTHROPIC_API_KEY` | *(empty)* | Anthropic credential. | +| `google_api_key` | `GOOGLE_API_KEY` | *(empty)* | Google credential, for `google-gla:*` / `google-vertex:*`. | +| `agent_thinking_budget` | `AGENT_THINKING_BUDGET` | *(unset)* | Optional token budget for Gemini extended reasoning. Unset disables it. | +| `agent_max_tool_calls` | `AGENT_MAX_TOOL_CALLS` | `10` | Tool calls allowed per session. | +| `agent_timeout_seconds` | `AGENT_TIMEOUT_SECONDS` | `120` | Wall-clock timeout wrapping one agent run. | +| `agent_retry_attempts` | `AGENT_RETRY_ATTEMPTS` | `3` | Retries on a failed model call. | +| `agent_retry_delay_seconds` | `AGENT_RETRY_DELAY_SECONDS` | `1.0` | Delay between retries. | +| `agent_session_ttl_minutes` | `AGENT_SESSION_TTL_MINUTES` | `120` | Session lifetime. | +| `agent_max_sessions_per_user` | `AGENT_MAX_SESSIONS_PER_USER` | `5` | Concurrent sessions per user. | +| `agent_enable_streaming` | `AGENT_ENABLE_STREAMING` | `true` | Enables token-by-token streaming over the WebSocket. | +| `agent_require_approval` | `AGENT_REQUIRE_APPROVAL` | `["create_alias","archive_run","save_scenario"]` | Tool names gated behind human approval. | +| `agent_approval_timeout_minutes` | `AGENT_APPROVAL_TIMEOUT_MINUTES` | `60` | How long a pending approval waits before expiring. | + +### The model identifier format + +`agent_default_model` and `agent_fallback_model` are validated on load by `validate_model_identifier`. The format is `provider:model-name`, where provider is one of `anthropic`, `openai`, `google-gla`, `google-vertex`, `ollama`. Three failures are rejected explicitly: + +- **No colon** — `claude-sonnet-4-5` alone is not a valid identifier. +- **Empty or blank model name** — `anthropic:` is rejected. +- **A nested provider prefix** — `google-gla:google-gla:gemini-3-flash` is rejected with a suggested correction, because it otherwise fails only later as a 404 at request time. Multi-colon Ollama tags stay valid: `ollama:llama3.1:8b` is fine, because `llama3.1` is not a provider name. + +Choosing an `ollama:` model runs the agent fully locally with no API key. + +### `agent_require_approval` is a safety boundary + +This list is the human-in-the-loop gate: every tool named in it pauses the agent and waits for an explicit approval before running. It ships with the three mutating tools — `create_alias`, `archive_run`, `save_scenario`. Removing a name from this list lets an agent perform that mutation **without asking anyone**. Per [AGENTS.md](../../AGENTS.md), widening an agent's mutation surface requires adding the new tool name here. See [Chat and knowledge](analyst/chat-and-knowledge.md). + +Note the format: it is parsed as a **JSON array**, so `.env` needs `AGENT_REQUIRE_APPROVAL=["create_alias","archive_run","save_scenario"]` — not a comma-separated bare string. + +## Seeder + +| Field | Env var | Default | What it controls | +|---|---|---|---| +| `seeder_default_seed` | `SEEDER_DEFAULT_SEED` | `42` | Random seed. Same seed plus same scenario reproduces the same dataset. | +| `seeder_default_stores` | `SEEDER_DEFAULT_STORES` | `10` | Stores generated by default. | +| `seeder_default_products` | `SEEDER_DEFAULT_PRODUCTS` | `50` | Products generated by default. | +| `seeder_batch_size` | `SEEDER_BATCH_SIZE` | `1000` | Insert batch size. | +| `seeder_enable_progress` | `SEEDER_ENABLE_PROGRESS` | `true` | Emits progress events during generation. | +| `seeder_allow_production` | `SEEDER_ALLOW_PRODUCTION` | `false` | Whether seeding is permitted when `app_env` is `production`. | +| `seeder_require_confirm` | `SEEDER_REQUIRE_CONFIRM` | `true` | Requires explicit confirmation for destructive seeder operations. | + +`seeder_allow_production` and `seeder_require_confirm` are the two guards that keep a synthetic-data generator from overwriting a dataset someone cares about. See [Seeding data](operator/seeding-data.md). + +## Runtime-editable settings (no restart) + +The AI-model settings are the deliberate exception to the restart rule. The **`/admin` → AI Models** tab persists overrides in an `app_config` database table and re-applies them onto the live `Settings` singleton — the same mechanism runs at startup, so an override survives a restart too. + +What is editable live: the agent model, the RAG embedding provider/model/dimension, and provider API keys. The endpoints are `GET /config/ai`, `PATCH /config/ai`, `GET /config/providers/health`, and `GET /config/ollama/models`. + +**API keys are always masked on read.** `GET /config/ai` never returns a key value. + +## Next + +- [Troubleshooting](troubleshooting.md) — when a setting is not doing what you expect. +- [Running the stack](operator/running-the-stack.md) — how host mode and Compose mode differ. diff --git a/docs/manual/faq.md b/docs/manual/faq.md new file mode 100644 index 00000000..8461911c --- /dev/null +++ b/docs/manual/faq.md @@ -0,0 +1,100 @@ +# FAQ + +Short answers to the questions this system actually gets, each linking to the chapter with the full story. + +**Purpose:** fast orientation for questions that don't need a whole chapter. +**Intended reader:** everyone; skimmable. + +## What this is + +**Is this a real forecasting product I can point at my business?** +No. ForecastLabAI is a portfolio-grade system that exercises the *complete* forecasting lifecycle honestly, on one machine, with synthetic data. The engineering is real — leakage controls, reproducible runs, artifact integrity, an audited promotion gate. The data is generated. Pointing it at real data would mean replacing the seeder with real ingest and re-validating every measurement ([What ForecastLabAI is](operator/concepts.md)). + +**Are the accuracy numbers meaningful?** +They are real measurements of model behavior on generated data — reproducible from a seed, and correct as measurements. They are **not** evidence about real retail demand. A model that wins here has won on patterns the generator created ([Backtesting](analyst/backtesting.md)). + +**Do I need a cloud account?** +No. Everything runs on a single host by design. Adding a managed-cloud SDK to the `app/` core path is explicitly forbidden in [AGENTS.md](../../AGENTS.md) — it would violate the single-host vision. + +**Do I need a GPU?** +No. The forecasting models are scikit-learn, LightGBM, and XGBoost on CPU. The only GPU path is optional: a Compose profile that runs **Ollama** on GPU for local embeddings and local agent models ([Running the stack](operator/running-the-stack.md)). + +**Do I need an OpenAI or Anthropic key?** +Only for the chat agents and OpenAI-backed embeddings. Forecasting, backtesting, the registry, the Explorer, and every analytical dashboard page work with no key at all. Embeddings can run fully local through Ollama. + +## Running it + +**Why is everything zero after installing?** +A fresh database is empty and nothing seeds itself. Run `make demo`, or generate data from **Admin → Data seeding** ([Quickstart](operator/quickstart.md)). + +**Why port 5433 for Postgres?** +So the Compose database does not collide with a Postgres you may already run on 5432. The container still listens on 5432 internally — 5433 is the host-side publication ([Running the stack](operator/running-the-stack.md)). + +**Why does the database URL differ between host mode and Compose mode?** +Different networks. From the host it is `localhost:5433`; from inside the Compose network it is `postgres:5432`. The backend container sets `DATABASE_URL` itself, overriding `.env`, so the `.env` value can stay the host-mode default ([Configuration reference](configuration.md)). + +**Is the seeded data the same every time?** +Yes, if the seed and scenario are the same. `seeder_default_seed` is 42. That reproducibility is the point — it lets two runs be compared honestly ([Seeding data](operator/seeding-data.md)). + +**Can I safely re-send the same sales batch?** +Yes. `POST /ingest/sales-daily` resolves natural keys and upserts idempotently ([API reference](integrator/api-reference.md)). + +## Models and results + +**How many models are there?** +Eleven model types in three families: five baselines, four tree models, two additive models. The family is computed from the model type, never stored ([Forecasting](analyst/forecasting.md)). + +**Why keep such simple baselines around?** +As honest comparison points. A machine-learning model that cannot beat "same day last week" is not worth deploying, and without the baseline you would not know. + +**Which metric decides the winner?** +WAPE, with a fixed tie-break chain: WAPE → sMAPE → |bias| → MAE. WAPE is scale-free *and* stable at low volume, which sMAPE is not ([Backtesting](analyst/backtesting.md)). + +**Positive bias — is that good?** +No. **Positive bias means the model under-forecasts**, which risks stockouts. Negative bias means it over-forecasts, which risks overstock ([Backtesting](analyst/backtesting.md)). + +**Does feature importance tell me what drives demand?** +No. It reflects how much each feature reduced the model's *training* error — correlation with the model's fit, not real-world causation. Two products with similar importance profiles need not share a business driver ([Forecasting](analyst/forecasting.md)). + +**Why can't my regression model just forecast forward?** +A feature-aware model needs a **future** feature frame — future prices, promotions, inventory. Rather than invent one, the system blocks the auto-forecast and routes you to the What-If Planner, where you state those assumptions explicitly ([Demand and planning](analyst/demand-and-planning.md)). + +**Why won't these two runs compare?** +Comparability requires all three: same grain, overlapping data windows, same feature-frame version. Anything else is not an apples-to-apples comparison ([Champion selector](analyst/champion-selector.md)). + +**Is the safety-stock number a real inventory optimisation?** +No, and the UI says so. It is a labelled deterministic heuristic over demand variability with a constant lead time. It **never** influences model ranking ([Champion selector](analyst/champion-selector.md)). + +## Promotion and safety + +**Can the system promote a model automatically?** +No. The app *recommends*; a human *approves*; the decision is recorded with the approver, the reason, and whether it overrode the recommendation ([Champion selector](analyst/champion-selector.md)). + +**Can I promote a model that scores worse than the champion?** +Yes, deliberately — but you must tick an explicit acknowledgement showing the exact deltas. What you **cannot** override is a failed artifact verification: that gate has no checkbox ([Champion selector](analyst/champion-selector.md)). + +**Can the chat agent change my registry behind my back?** +No. Every mutating tool is in `agent_require_approval` — `create_alias`, `archive_run`, `save_scenario` — and pauses for a person. Removing a name from that list is what would make it possible, which is why widening the list is a gated decision ([Chat and knowledge](analyst/chat-and-knowledge.md)). + +**What happens if I ignore an approval prompt?** +It expires after `agent_approval_timeout_minutes` (60) and the tool does not run. + +## Building on it + +**Where is the authoritative API contract?** +The interactive OpenAPI schema at `http://localhost:8123/docs`. It is generated from the code, so it is always current; this manual is written by hand and explains *why* rather than restating every field ([API reference](integrator/api-reference.md)). + +**Why do errors look like `application/problem+json`?** +RFC 7807, used uniformly. Ad-hoc error shapes and bare `HTTPException` with raw strings are forbidden by repository rules, so one parser handles every error from every endpoint. + +**Why can't one feature slice import another?** +It is the rule that keeps the 19 slices independent and the import graph one-way (`app/features/* → app/shared`). Cross-cutting code goes through `app/core/` or `app/shared/`. The `ModelFamily` enum was moved to `app/shared/` for exactly this reason ([Code architecture](integrator/code-architecture.md)). + +**Can I edit an existing Alembic migration?** +No. Migrations are forward-only once merged — add a new one ([Extending ForecastLabAI](integrator/extending.md)). + +**Why does adding an unknown model type not crash the dashboard?** +`model_family_for` classifies unknown types as `baseline` and logs a warning, so a model added before the taxonomy map is updated degrades gracefully instead of raising ([Forecasting](analyst/forecasting.md)). + +**Which checks must pass before I commit?** +Ruff (check and format), mypy `--strict`, pyright `--strict`, and the non-integration pytest suite. All of them gate merge ([CI and quality gates](integrator/ci-and-quality-gates.md)). diff --git a/docs/manual/glossary.md b/docs/manual/glossary.md new file mode 100644 index 00000000..3f585da0 --- /dev/null +++ b/docs/manual/glossary.md @@ -0,0 +1,132 @@ +# Glossary + +The vocabulary this manual uses, defined once and used consistently. + +**Purpose:** settle what each term means in *this* system, especially where a general ML word has a specific local meaning. +**Intended reader:** everyone; skimmable and cross-linked. + +## Retail and demand + +**SKU** — a stock-keeping unit; one row in the `product` table. This manual uses "product" and "SKU" interchangeably. + +**Grain** — the level a forecast is made at. In ForecastLabAI the grain is always a **(store, product) pair** over daily time steps. Two model runs are only comparable if they share a grain. + +**Horizon** — how many days ahead a forecast predicts. Bounded by `forecast_max_horizon` (default 90); the default is `forecast_default_horizon` (14). + +**Lead time** — the days between placing a replenishment order and receiving it. An input to the safety-stock heuristic, not something the models predict. + +**Safety stock** — buffer inventory held to absorb demand variability over the lead time. Computed here by a deterministic, clearly-labelled heuristic — see [Champion selector](analyst/champion-selector.md). + +**Reorder point** — expected demand over the lead time plus safety stock; the level at which you should reorder. + +**Exogenous signal** — a driver outside the sales history itself (weather, macro indicators). Stored in `exogenous_signal` and exposed as optional V2 feature packs. + +## Data and features + +**Feature** — a model-ready column derived from raw history: a lag, a rolling statistic, a calendar flag, a price level. + +**Leakage** — letting information from the future reach a model that is supposed to predict the future. It inflates accuracy and invalidates the measurement. Prevented structurally with `shift(lag)` and `shift(1).rolling()` patterns, and locked by `app/features/featuresets/tests/test_leakage.py`, which is the spec. + +**Time-safe** — a feature computation that provably cannot leak. The word appears throughout the API and this manual with that exact meaning. + +**Feature frame** — the versioned contract describing which columns a model consumes. +- **V1** — target-only: lags plus same-day-of-week mean. Every model can train on V1. It is the backend default. +- **V2** — feature-aware: the richer contract, with eleven optional feature packs. Available to the tree and additive families only. + +**Feature pack** — a named subset of V2 columns you can toggle on or off (`calendar`, `price_promo`, `inventory`, and so on). See [Forecasting](analyst/forecasting.md). + +**Feature safety class** — a per-pack chip: `Safe`, `Conditionally safe`, or `Requires supplied data`. The last one means the pack reads a column your production pipeline must keep populated. + +**Cutoff date** — the date a feature computation treats as "now". Nothing after it may influence a feature. + +## Models + +**Model type** — one of the eleven concrete forecasters (`naive`, `regression`, `prophet_like`, …). The full list is in [Forecasting](analyst/forecasting.md). + +**Model family** — the computed grouping of a model type into `baseline`, `tree`, or `additive`. Derived from `model_type` by `app/shared/model_taxonomy.py`; **never stored in the database**, and surfaced on API responses as a computed field. An unknown model type classifies as `baseline` so a newly added model does not break the dashboard. + +**Baseline** — a deliberately simple forecaster (last value, same day last week, moving average). Baselines exist as honest comparison points: a machine-learning model is only worth using if it beats them. + +**Feature-aware model** — a tree- or additive-family model that consumes a feature frame. These cannot auto-forecast forward without a future feature frame — see the capability limit in [Champion selector](analyst/champion-selector.md). + +## Measurement + +**Backtest** — replaying history with time-series cross-validation to measure how accurate a model *would have been*. + +**Fold** — one train/test split within a backtest. Splits are **rolling** (fixed-width training window) or **expanding** (training window grows). + +**MAE** — mean absolute error, in units. Interpretable, but not comparable across SKUs of different volume. + +**sMAPE** — symmetric mean absolute percentage error. Scale-free, but unstable when actual demand is near zero. + +**WAPE** — weighted absolute percentage error: total absolute error divided by total actual demand. **The default ranking metric**, because it is scale-free *and* stable at low volumes. + +**Bias** — signed average error. **Positive bias means the model under-forecasts** (stockout risk); negative means it over-forecasts (overstock risk). + +**RMSE** — root mean squared error; penalises large misses more than MAE. + +**Stability** — how consistent a model's error is across folds. A model that is accurate on average but wild fold-to-fold is a risk. + +**Horizon bucket** — error grouped by forecast distance: `h_1_7`, `h_8_14`, `h_15_28`, `h_29_plus`. Near-term and far-term accuracy are different questions. + +## Registry and lifecycle + +**Model run** — one training execution, tracked in `model_run` with its config, metrics, and artifact. Moves through `pending → running → success` (or `failed`). + +**Artifact** — the serialized fitted model on disk, with a SHA-256 checksum the registry can re-verify on demand. + +**Alias** — a human-friendly, movable pointer to one successful run (`production`, `champion`). Stored in `deployment_alias`. An alias may point only to a successful run. + +**Stale alias** — an alias the system flags as possibly out of date, with a reason: a newer successful run exists, the artifact failed verification, the target run is no longer successful, or the feature-frame version drifted. + +**Promotion** — pointing an alias at a run. Approval-gated and audited: it records who approved it, the reason, and whether it overrode the recommendation. **Never automatic.** + +**Champion / challenger** — the currently promoted model versus a candidate competing to replace it. Two runs are comparable only if they share a grain, have overlapping data windows, and use the same feature-frame version. + +**Override** — training or promoting a model that was *not* the ranked recommendation. Flagged `is_override=true` and audited. + +## Jobs and batches + +**Job** — an asynchronous unit of work (`train`, `predict`, or `backtest`), tracked in `job` with status and result JSON. + +**Batch** — a matrix of jobs submitted together, expanded across (store, product) pairs × model configs. Bounded by `batch_max_scope_expansion`. + +**Sweep preset** — a named, prefilled batch matrix (for example "Quick baseline sweep"). + +**Drain** — the wait when cancelling a batch or selection run, while in-flight fits finish. A scikit-learn or LightGBM fit **cannot be cancelled mid-call**, which is why the drain has a timeout. + +## Agents and RAG + +**RAG** — retrieval-augmented generation: answering from indexed documents rather than model memory alone. + +**Chunk** — a passage a document is split into for embedding. Markdown splits by heading; OpenAPI splits by endpoint. + +**Embedding** — the numeric vector representing a chunk's meaning, stored in Postgres via **pgvector**. + +**Cosine similarity** — how retrieval scores a chunk against a query. Results below `rag_similarity_threshold` (default 0.7) are not returned. + +**Agent** — a PydanticAI conversational assistant. Two types: **`rag_assistant`** (answers from the knowledge base) and **`experiment`** (can run forecasting experiments). + +**Tool** — a typed function an agent may call. Read-only tools run immediately. + +**Human-in-the-loop (HITL) approval gate** — the mechanism that pauses an agent before a *mutating* tool runs and waits for a person. The gated set is `agent_require_approval` — by default `create_alias`, `archive_run`, `save_scenario`. + +**Session** — one conversation, bounded by a token budget, a tool-call cap, and a timeout. + +## Scenarios + +**Scenario plan** — a saved set of what-if assumptions (price, promotion, holiday, inventory, lifecycle) applied to an existing forecast. Stored in `scenario_plan`. + +**`model_exogenous`** — the scenario method that genuinely **re-forecasts** through the assumptions using a regression baseline. + +**Heuristic adjustment** — the fallback scenario method: a deterministic, clearly-labelled adjustment rather than a real re-forecast. The dashboard always shows which of the two produced a result. + +## System shape + +**Vertical slice** — a self-contained feature directory under `app/features//` holding its own models, schemas, service, routes, and tests. There are **19**. A slice may not import another slice; shared code lives in `app/core/` or `app/shared/`. + +**Problem details** — the RFC 7807 `application/problem+json` error envelope every endpoint uses. See [API reference](integrator/api-reference.md). + +**The Forge** — the synthetic data seeder. Everything it produces is generated, reproducible from a seed, and **not real retail data**. + +**Showcase** — the `/showcase` page that runs the whole pipeline live in the browser as streamed status cards. diff --git a/docs/manual/integrator/api-reference.md b/docs/manual/integrator/api-reference.md new file mode 100644 index 00000000..5ba3b14c --- /dev/null +++ b/docs/manual/integrator/api-reference.md @@ -0,0 +1,196 @@ +# API reference + +The conventions every endpoint shares, the error envelope, and a map of all twenty endpoint groups. + +**Purpose:** enough to call the API confidently and handle its failures correctly. +**Intended reader:** integrators building against ForecastLabAI. + +## What you'll accomplish + +A working mental model of the API surface, and the ability to parse any error it returns with one code path. + +## The authoritative contract is generated + +The OpenAPI schema at **http://localhost:8123/docs** is generated from the code and is therefore always current. **It is the contract.** This chapter is written by hand and deliberately does not restate every field — it explains the conventions, the error semantics, and where each capability lives, so you can read the generated schema quickly. + +Where this chapter and `/docs` disagree, `/docs` is right. + +## Shared conventions + +**Validation is Pydantic v2 at every boundary.** Malformed input fails at the edge with a field-level error, never deeper. + +**Errors are RFC 7807** `application/problem+json`, uniformly. No endpoint returns an ad-hoc error shape; bare `HTTPException` with a raw string is forbidden by repository rules. One parser handles every error from every endpoint. + +**Long work is asynchronous.** Training, prediction, backtesting, batches, and champion comparisons return a handle and are polled. Nothing blocks an HTTP connection on a model fit. + +**Requests carry a correlation ID.** `RequestIdMiddleware` assigns one, and it appears in structured logs and in error bodies as `request_id` — the join key between a client failure and its server-side log lines. + +**Ingest is idempotent.** `POST /ingest/sales-daily` resolves natural keys (store code, SKU) to IDs and upserts, so re-sending a batch is safe. + +## The error envelope + +```json +{ + "type": "/errors/unprocessable-entity", + "title": "Unprocessable Entity", + "status": 422, + "detail": "…specific to this occurrence…", + "instance": "…", + "code": "UNPROCESSABLE_ENTITY", + "request_id": "…" +} +``` + +`type` is a stable URI under `/errors` — **switch on it**, not on prose. `title` is stable per type; `detail` is specific to the occurrence. `errors` carries field-level detail on 422 validation failures. The model allows extensions, per RFC 7807. + +### The type registry + +| `type` | Typical status | Meaning | +|---|---|---| +| `/errors/validation` | 422 | The **input** failed validation. | +| `/errors/unprocessable-entity` | 422 | Input was well-formed; the **state** forbids the action. | +| `/errors/bad-request` | 400 | The request does not apply to this resource. | +| `/errors/not-found` | 404 | No such resource. | +| `/errors/conflict` | 409 | Conflicts with existing state. | +| `/errors/unauthorized` | 401 | — | +| `/errors/forbidden` | 403 | — | +| `/errors/rate-limited` | 429 | — | +| `/errors/gateway-timeout` | 504 | A drain or upstream wait timed out. | +| `/errors/service-unavailable` | 503 | — | +| `/errors/embedding-auth` | — | The embedding provider rejected the credential. | +| `/errors/agent-fallback-exhausted` | — | Primary **and** fallback agent models both failed. | +| `/errors/database` | 500 | Database-level failure. | +| `/errors/internal` | 500 | Unhandled server error. | + +### The distinction that matters most + +`validation` and `unprocessable-entity` are **both 422 and deliberately different**: + +- **`/errors/validation`** — *you sent something wrong.* Fix the request. +- **`/errors/unprocessable-entity`** — *what you sent cannot be done right now.* The request is fine; the state is not. + +Promoting an untrained model is `unprocessable-entity`; promoting with a malformed alias name is `validation`. Retrying the first after training makes sense; retrying the second unchanged never will. Client retry logic should branch here. + +Similarly, `bad-request` (400) marks a request that does not apply at all — asking a baseline run for feature importances, for instance. It is permanent, not transient. + +## Endpoint groups + +Twenty routers. `/health` sits at the root; the rest are prefixed. + +### Platform + +| Prefix | Purpose | +|---|---| +| `/health` | Liveness probe → `{"status":"ok"}`. | +| `/ingest` | Batch sales load — idempotent. | +| `/dimensions` | Stores and products: list with filters, search, sorting, pagination; fetch by id. | +| `/analytics` | Read-only aggregates: `kpis`, `drilldowns`, `timeseries`, `inventory-status`. | +| `/seeder` | The Forge — see [Seeding data](../operator/seeding-data.md). | +| `/config` | Runtime AI configuration; keys always masked on read. | + +Analytics responses are bounded by `analytics_max_rows` (10000) and `analytics_max_date_range_days` (730). + +### Modelling + +| Prefix | Purpose | +|---|---| +| `/featuresets` | `compute` and `preview` time-safe features up to a cutoff. | +| `/forecasting` | `train`, `predict`, and feature metadata by run or job. | +| `/backtesting` | `run` — rolling or expanding cross-validation. | +| `/model-selection` | The champion selector workflow. | +| `/explain` | Forecast explainability. Note the prefix is `/explain`, not `/explainability`. | +| `/scenarios` | What-if simulation, saved plans, and multi-plan comparison. | + +### Orchestration + +| Prefix | Purpose | +|---|---| +| `/jobs` | Submit, list, inspect, and cancel `train` / `predict` / `backtest` jobs. | +| `/batch` | Matrix submissions with bounded concurrency. | +| `/registry` | Runs, comparison, artifact verification, and aliases. | +| `/ops` | `summary`, `retraining-candidates`, `model-health`. | +| `/demo` | `run` the pipeline, and a WebSocket event stream. | + +### Conversational + +| Prefix | Purpose | +|---|---| +| `/rag` | Index, retrieve, list, and delete knowledge sources. | +| `/agents` | Sessions, chat, approval, plus `WS /agents/stream`. | + +## Worked flows + +### Train, then check on it + +```bash +# Submit +curl -X POST http://localhost:8123/jobs \ + -H 'Content-Type: application/json' \ + -d '{"job_type": "train", ...}' +# → {"job_id": "..."} + +# Poll +curl http://localhost:8123/jobs/{job_id} +``` + +A job carries status, result JSON, error detail, and the run it produced. + +### Verify an artifact before trusting a run + +```bash +curl http://localhost:8123/registry/runs/{run_id}/verify +``` + +Re-computes the artifact's SHA-256 against the recorded value. This is the check the promotion gate runs, and the one gate with no operator override — see [Artifacts and the registry](artifacts-and-registry.md). + +### Compare two runs + +```bash +curl http://localhost:8123/registry/compare/{run_id_a}/{run_id_b} +``` + +Comparability requires the same grain, overlapping data windows, and the same `feature_frame_version`. + +### The champion workflow + +``` +POST /model-selection/runs → 202 + monitor URL +GET /model-selection/{id} → poll to terminal +POST /model-selection/{id}/train-winner (or /train-selected) +POST /model-selection/{id}/predict +POST /model-selection/{id}/promote +``` + +`promote` requires a valid alias name, an `approved_by`, `acknowledge_non_recommended=true` for an override, and a trained model — each a distinct 422. See [Champion selector](../analyst/champion-selector.md). + +## Behaviors worth designing for + +**A 504 from a cancel is not a failed cancel.** `DELETE /batch/{id}` and `DELETE /model-selection/{id}` drain in-flight work first. scikit-learn and LightGBM fits cannot be cancelled mid-call, so the drain is bounded by a timeout (default 30s) and exceeding it returns 504. The cancellation is still in progress — re-poll rather than re-issuing. + +**Batches are rejected, not truncated.** Expanded scope over `batch_max_scope_expansion` (1000) fails at submission. + +**Feature-aware models refuse to auto-forecast.** `POST /model-selection/{id}/predict` blocks for `regression`, `prophet_like`, `lightgbm`, `xgboost`, and `random_forest` — they need a future feature frame. Use `/scenarios` instead. This is a designed refusal, not a bug to work around. + +**V1 plus `feature_groups` is a 422.** Feature packs are V2-only. + +**Feature importance has three distinct failure codes.** 400 (baseline — no learned importance), 404 (unknown run/job), 422 (no artifact yet, artifact deleted, missing `ml-*` extra at unpickle, or an estimator without `feature_importances_` — which includes `regression`'s `HistGradientBoostingRegressor`). + +**Model family is computed, never stored.** It arrives as a computed field derived from `model_type`; unknown types classify as `baseline` and log a warning rather than raising. + +## WebSockets + +| Endpoint | Streams | +|---|---| +| `WS /agents/stream` | Agent tokens and tool-call events, including `approval_required`. | +| `WS /demo/stream` | Per-step demo pipeline events for the Showcase page. | + +An `approval_required` event means the agent has **stopped** and will not proceed until `POST /agents/sessions/{id}/approve` resolves it, or it expires after `agent_approval_timeout_minutes`. + +## No authentication + +There is none. The API is unauthenticated and intended for single-host local use — consistent with the system's scope ([What ForecastLabAI is](../operator/concepts.md#what-this-is-not)). `api_host` defaults to `0.0.0.0`, which listens on all interfaces; binding it to a reachable network exposes an unauthenticated API, and that is your deliberate choice. + +## Next + +- [Code architecture](code-architecture.md) — where each endpoint group lives. +- [Artifacts and the registry](artifacts-and-registry.md) — the integrity contract. diff --git a/docs/manual/integrator/artifacts-and-registry.md b/docs/manual/integrator/artifacts-and-registry.md new file mode 100644 index 00000000..c1c71900 --- /dev/null +++ b/docs/manual/integrator/artifacts-and-registry.md @@ -0,0 +1,138 @@ +# Artifacts and the registry + +How a trained model becomes a verifiable, promotable thing — and what the system refuses to do when it cannot verify one. + +**Purpose:** understand run lifecycle, artifact integrity, and alias semantics well enough to build on them. +**Intended reader:** integrators consuming registry data or automating promotion. + +## What you'll accomplish + +The ability to trace any served prediction back to a specific run, its configuration, its data window, and a checksum that proves the artifact has not changed. + +## Why a registry exists + +A trained model is useless as evidence unless you can say *which* model it was. Without a registry you get a directory of pickle files and an argument about which one is in production. + +A `model_run` row answers: what model type, at what grain, over which data window, with which configuration and seed, scoring what, backed by which artifact — and whether that artifact still hashes to what it did when it was written. + +## The run lifecycle + +``` +pending → running → success + ↘ failed +``` + +| Endpoint | Purpose | +|---|---| +| `POST /registry/runs` | Create a run record — starts `pending`. | +| `GET /registry/runs` | List with filters, pagination, sorting. | +| `GET /registry/runs/{run_id}` | Details, metrics, runtime info. | +| `PATCH /registry/runs/{run_id}` | Update status, metrics, or artifact location. | +| `GET /registry/runs/{run_id}/verify` | **Verify artifact SHA-256.** | +| `GET /registry/compare/{a}/{b}` | Diff two runs. | + +Only a **successful** run may be aliased. A `failed` run keeps its record — a failure is data about what does not work, and deleting it would let the same experiment be repeated blindly. + +## Artifact integrity + +When a model is fitted, the artifact is written to disk and its **SHA-256** recorded on the run. + +`GET /registry/runs/{run_id}/verify` re-computes the hash and compares. Three outcomes matter: + +- **Verified** — the file on disk is byte-identical to what was recorded. +- **Mismatch** — the file changed. Whatever it is now, it is not what was measured. +- **Missing** — the file is gone. + +The last two are indistinguishable in consequence: the run's metrics describe a model you can no longer produce. + +### This is the one gate with no override + +The Promote dialog auto-fetches the verification result. **A failure disables the Promote button with no operator override** — unlike the worse-WAPE and feature-frame-mismatch gates, which are acknowledgeable checkboxes. + +The asymmetry is deliberate. "This model scores worse and I accept that" is a judgement a human can make. "This file is not the model I measured" is not a judgement at all — there is nothing to weigh. Re-train to produce a verifiable artifact. + +Artifact roots are configurable: `forecast_model_artifacts_dir`, `backtest_results_dir`, `registry_artifact_root`, and `showcase_export_root` — all under `./artifacts/` by default, and inside the `forecastlab_artifacts` named volume in container mode. + +**Deleting an artifact does not delete its run.** You get a row whose metrics still render, whose verification now fails, and whose feature-importance endpoint returns `422`. Archive runs through the registry rather than deleting files underneath it. + +## Aliases + +An alias is a movable, human-friendly pointer to one successful run. + +| Endpoint | Purpose | +|---|---| +| `POST /registry/aliases` | Create or move an alias. | +| `GET /registry/aliases` | List. | +| `GET /registry/aliases/{name}` | Fetch one. | +| `DELETE /registry/aliases/{name}` | Delete. | + +Names must match `^[a-z0-9][a-z0-9\-_]*$`. Re-pointing an existing name is an **upsert**, not an error — that is how promotion works. + +Aliases exist so a consumer can depend on `production` rather than a run id that changes every retrain. The indirection is the point: the consumer's contract stays stable while the model behind it moves. + +## Promotion is a recorded decision + +`POST /model-selection/{id}/promote` registers the trained model as a `model_run` transitioned to SUCCESS with a verified artifact, points an alias at it, and writes a `promotion_decision` audit: approver, alias, run id, decision, reason, and whether it overrode the recommendation. + +Four preconditions, each its own `422`: + +| Requirement | Rationale | +|---|---| +| Valid alias name | Aliases are addressable identifiers. | +| `approved_by` present | **Promotion is never anonymous.** | +| `acknowledge_non_recommended=true` for an override | Deliberate, not accidental. | +| The model is trained | You cannot promote what does not exist. | + +**Compare and promote are separate operations.** Promote performs no ranking — it registers and aliases an already-trained model. Keeping them apart is what makes the audit meaningful: the comparison is *evidence*, the promotion is a *decision*, and they are recorded as different things by different actors. + +Full workflow in [Champion selector](../analyst/champion-selector.md). + +## Comparability + +Two runs are comparable **if and only if**: + +1. Same grain (`store_id`, `product_id`). +2. Overlapping data windows. +3. Same `feature_frame_version` — absent on older runs, which default to V1. + +`GET /registry/compare/{a}/{b}` returns the diff, and the dashboard renders a **Champion compatibility** badge with the verdict plus a feature-frame-version row. + +If you automate promotion, check this verdict rather than comparing metrics directly. Two WAPE numbers from different grains are two answers to different questions, and nothing in the numbers themselves will tell you that. + +## Staleness + +`GET /ops/model-health` surfaces stale aliases with a reason: + +| Reason | Meaning | Urgency | +|---|---|---| +| `newer success run` | A newer successful run exists at this grain. | Routine. | +| `artifact not verified` | The alias's artifact failed verification. | **Urgent.** | +| `run not success` | The alias points at a failed or archived run. | High. | +| `V mismatch` | The newest comparable run uses a different `feature_frame_version`. | Subtle. | + +`V mismatch` is the one that does not announce itself. Nothing errors when the feature contract drifts — a downstream pipeline keeps running while supplying columns the new version does not expect, or omitting ones it does. It degrades quietly, which is why crossing that boundary requires an explicit acknowledgement at promotion time. + +## Duplicate policy + +`registry_duplicate_policy` (default `detect`): + +- `detect` — flag but allow. +- `deny` — reject. +- `allow` — record silently. + +`detect` surfaces accidental repeat work without blocking a deliberate re-run. + +## Building on the registry + +**Resolve through aliases, not run ids**, so retraining does not require a consumer change. + +**Verify before serving.** `GET /registry/runs/{run_id}/verify` is cheap next to serving predictions from an artifact that is not what you measured. + +**Treat `feature_frame_version` as part of the contract.** If you feed a promoted model, a version change means your input columns must change too. + +**Never write registry rows directly.** A hand-written row bypasses artifact bookkeeping — its metrics will display and its promotion will fail. See [Data model](data-model.md#querying-directly). + +## Next + +- [Extending ForecastLabAI](extending.md) — adding models and slices without breaking these guarantees. +- [CI and quality gates](ci-and-quality-gates.md) — what enforces all of this. diff --git a/docs/manual/integrator/ci-and-quality-gates.md b/docs/manual/integrator/ci-and-quality-gates.md new file mode 100644 index 00000000..0b9ed717 --- /dev/null +++ b/docs/manual/integrator/ci-and-quality-gates.md @@ -0,0 +1,118 @@ +# CI and quality gates + +What must pass before code merges, what runs in GitHub Actions, and how a release happens. + +**Purpose:** get a change through the pipeline without surprises. +**Intended reader:** integrators contributing code. + +## What you'll accomplish + +A local check sequence that matches CI, and an understanding of the branch, commit, and release conventions. + +## The local gates + +Run these before every commit. They mirror what CI enforces, so a green local run is a strong predictor of a green pipeline. + +```bash +uv run ruff check . && uv run ruff format --check . +uv run mypy app/ && uv run pyright app/ # both --strict +uv run pytest -v -m "not integration" +``` + +Frontend work adds: + +```bash +cd frontend && pnpm tsc --noEmit && pnpm lint && pnpm test --run +``` + +**Two type checkers, both strict, both gating.** That is deliberate rather than redundant: mypy and pyright disagree about enough real cases that passing both is a meaningfully stronger guarantee than passing either. + +Integration tests need a live database and are excluded from the fast loop: + +```bash +docker compose up -d +uv run pytest -v -m integration +``` + +They run against **real** Compose Postgres. Mocking the database in an integration test is forbidden — a mocked integration test proves nothing about the thing it names. + +## The CI workflow + +`.github/workflows/ci.yml` runs on push, pull request, and manual dispatch, with in-progress runs cancelled when a newer commit lands on the same ref. + +| Job | Runs | +|---|---| +| **Lint & Format** | `ruff check` and `ruff format --check`. | +| **Type Check** | `mypy app/` then `pyright app/`. | +| **Test** | Migrations against a service Postgres, then the suite. | +| **Migration Check** | Applies migrations to a **fresh** database, then verifies no pending migrations remain. | + +Dependencies install with `uv sync --frozen --all-extras --dev` — `--frozen` means the lockfile is authoritative, so a `uv.lock` that drifted from `pyproject.toml` fails CI rather than silently resolving something new. + +### Migration Check is the subtle one + +It does two things a normal test run does not: it proves migrations apply cleanly **from empty**, and it proves the ORM models and the migration history agree — that autogenerate would produce nothing new. + +That second check catches the common mistake of changing a `models.py` and forgetting the migration. Locally everything works, because your database already has the column from a previous manual step. On a fresh database it does not exist. + +## Other workflows + +| Workflow | Trigger | Purpose | +|---|---|---| +| `cd-release.yml` | push to `main` | Release automation. | +| `e2e-nightly.yml` | nightly cron (07:00 UTC) + dispatch | Full end-to-end run. | +| `schema-validation.yml` | changes to `alembic/**`, `**/models.py`, `core/database.py` | Path-scoped schema checks. | +| `dependency-check.yml` | weekly cron (Sun 00:00 UTC) + dispatch | Dependency review. | +| `phase-snapshot.yml` | push to `phase-*` branches | Phase snapshots. | + +The nightly e2e is what exercises the full pipeline against a real stack — the same ground `make demo` covers locally. + +## Branches and commits + +**Branches** — `/` off `dev`; `hotfix/*` off `main`. One branch per issue. + +**Commits** — `type(scope): description (#issue)`: + +- `type` ∈ `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `release` +- `scope` from the allow-list in `.claude/rules/commit-format.md` +- lowercase description, no trailing period +- **every commit references an open GitHub issue** + +**No AI co-author or "Generated with" trailer.** A hook enforces this — a commit carrying one is rejected. + +## The flow + +``` +branch off dev → implement → run the gates → PR into dev → CI green → merge +``` + +Releasing: + +``` +PR dev → main → release-please opens a Release PR → merging it tags vX.Y.Z +``` + +**release-please owns tagging.** Do not push tags by hand. Versioning is SemVer, and the project is pre-1.0 — which means `feat:` commits produce **PATCH** bumps, not MINOR. That surprises people who expect standard SemVer behavior; it is correct for a 0.x line. + +Never `git push --force` on `dev` or `main`. + +## Stop and ask before + +- Cutting `dev` → `main`, or pushing any tag. +- Bumping pydantic-ai, FastAPI, or SQLAlchemy major versions. +- Widening an agent's mutation surface without adding the tool to `agent_require_approval`. + +## When CI fails and local passed + +**`uv sync --frozen` failed** — `uv.lock` is out of step with `pyproject.toml`. Re-lock and commit the lockfile. + +**Migration Check failed but tests passed** — you changed a model without a migration. Your local database already had the column. + +**pyright failed where mypy passed** — the checkers disagree; both must pass. Usually a narrowing or overload case. + +**An integration test failed only in CI** — it depends on state a previous local run left behind. Reproduce with a clean database: `docker compose down -v && docker compose up -d`. + +## Next + +- [Extending ForecastLabAI](extending.md) — what to build, and what not to touch. +- [AGENTS.md](../../../AGENTS.md) — the full rule set these gates enforce. diff --git a/docs/manual/integrator/code-architecture.md b/docs/manual/integrator/code-architecture.md new file mode 100644 index 00000000..a7018155 --- /dev/null +++ b/docs/manual/integrator/code-architecture.md @@ -0,0 +1,125 @@ +# Code architecture + +A tour of `app/`: nineteen vertical slices, the shared layers beneath them, and the one import rule that holds it together. + +**Purpose:** find the code that owns a behavior, and understand why the boundaries are where they are. +**Intended reader:** integrators reading or extending the backend. + +## What you'll accomplish + +The ability to locate any capability in the tree, and to add code without violating the structure. + +## The shape + +``` +app/ +├─ main.py FastAPI app: lifespan, middleware, router wiring +├─ core/ cross-cutting infrastructure +├─ shared/ domain code shared across slices +└─ features/ 19 vertical slices +``` + +## Vertical slices + +Every domain lives under `app/features//` and owns its full stack: + +``` +app/features// +├─ models.py SQLAlchemy ORM (Mapped[] + mapped_column()) +├─ schemas.py Pydantic v2 request/response contracts +├─ service.py business logic +├─ routes.py the HTTP surface +└─ tests/ the slice's tests +``` + +The nineteen slices: + +| Group | Slices | +|---|---| +| **Platform** | `data_platform`, `ingest`, `dimensions`, `analytics`, `seeder`, `config` | +| **Modelling** | `featuresets`, `forecasting`, `backtesting`, `model_selection`, `explainability`, `scenarios` | +| **Orchestration** | `jobs`, `batch`, `registry`, `ops`, `demo` | +| **Conversational** | `rag`, `agents` | + +Routers are wired in `app/main.py` — twenty `include_router` calls, since `agents` contributes both a REST router and a WebSocket router. + +## The one rule + +> **A slice may not import from another slice.** + +Cross-cutting code goes through `app/core/` or `app/shared/`. The import graph is one-way: `app/features/* → app/shared/` and `app/features/* → app/core/`, never sideways. + +This is the constraint that keeps nineteen slices legible. Without it, `forecasting` reaches into `registry`, `registry` reaches back into `forecasting`, and within a few features the system has no boundaries left to reason about. + +### What the rule actually prevents — a real example + +`ModelFamily` and `model_family_for` originally lived in the `forecasting` slice. But `registry.schemas` needed `ModelFamily` at module scope for the `RunResponse.model_family` computed field. + +That single cross-slice import forced lazy-import workarounds across the registry boundary — because the eager import created a `forecasting ↔ registry` cycle that broke Alembic cold-boot. The fix was to move the enum to `app/shared/model_taxonomy.py`, restoring the one-way graph. + +The lesson generalises: when two slices need the same domain type, the type belongs in `app/shared/`, not in whichever slice defined it first. + +## `app/core/` — infrastructure + +| Module | Owns | +|---|---| +| `config.py` | The `Settings` class and the cached `get_settings()`. | +| `database.py` | Async engine and session maker. | +| `exceptions.py` | The `ForecastLabError` hierarchy. | +| `problem_details.py` | The RFC 7807 envelope and the `type` URI registry. | +| `logging.py` | structlog configuration and `get_logger`. | +| `middleware.py` | `RequestIdMiddleware`. | +| `health.py` | The unprefixed `/health` router. | + +`exceptions.py` defines the domain error hierarchy — `NotFoundError`, `ValidationError`, `DatabaseError`, `ConflictError`, `BadRequestError`, `UnprocessableEntityError`, `GatewayTimeoutError`, `EmbeddingProviderAuthError`, `AgentFallbackExhaustedError` — and `problem_details.py` maps each to a stable `type` URI. Raising a domain exception anywhere produces a correctly-shaped problem response without the route knowing anything about HTTP error formatting. + +## `app/shared/` — shared domain code + +| Module | Owns | +|---|---| +| `model_taxonomy.py` | `ModelFamily`, `model_family_for`, `KNOWN_MODEL_TYPES`. | +| `feature_frames/` | The V2 feature contract: `FeatureGroup`, group ordering, column manifests. | +| `seeder/` | The Forge: scenario presets, generators, config. | + +`KNOWN_MODEL_TYPES` is a public allow-list derived from the canonical family map, so a slice that must validate a `model_type` (the `demo` slice, for instance) can check membership without importing a sibling slice. It is derived rather than duplicated — it cannot drift — and a test locks that. + +## `app/main.py` — composition + +The application factory: configures logging, re-applies persisted runtime config overrides onto the `Settings` singleton, registers exception handlers, adds CORS and `RequestIdMiddleware`, and wires every router. + +The startup override re-application is what makes `/admin` → AI Models changes survive a restart: the same mechanism that applies a live change also replays stored overrides at boot. + +## Conventions + +**ORM** — SQLAlchemy 2.0 with `Mapped[]` and `mapped_column()`, async sessions throughout. + +**Validation** — Pydantic v2 at every boundary: HTTP, agent tools, seeder config. + +**Configuration** — always `get_settings()`. **Never `os.environ` in feature code.** That rule is what makes the [configuration reference](../configuration.md) a complete list rather than a best guess. + +**Paths** — `pathlib.Path`, never `os.path`. + +**Errors** — raise a domain exception from `app/core/exceptions.py`. Never a bare `HTTPException` with a raw string; never an ad-hoc error shape. + +**Migrations** — every schema change ships an Alembic migration, and migrations are **forward-only once merged**. + +**Time-safety** — feature engineering must prevent leakage: `shift(lag)`, `shift(1).rolling()`, entity-aware `groupby`. `app/features/featuresets/tests/test_leakage.py` is the specification, and weakening it is forbidden. + +## Two flows through the package + +**A forecast request:** `main.py` (startup, once) → `RequestIdMiddleware` assigns the correlation id → `forecasting/routes.py` → `schemas` validates → `forecasting/service.py` → the fitted artifact → `schemas` validates the response → structured log event. + +**A champion selection:** `model_selection/routes.py` accepts and returns `202` → the async runner backtests each candidate under a global concurrency bound → results rank by WAPE with the fixed tie-break chain → `train-winner` or `train-selected` fits the chosen model → `promote` writes a `model_run` through the `registry` slice's *HTTP-independent* service boundary and records a `promotion_decision` audit. + +Note the asymmetry in the second flow: sequencing lives in the slice's service, while the modules it calls stay individually testable. That is the same division the demo script makes at a larger scale — `scripts/run_demo.py` drives only the published HTTP surface and never imports `app.features.*`, so drift between the deployed API and its runtime behavior shows up as a real failure rather than passing silently. + +## Frontend + +`frontend/` is React 19 + TypeScript with Vite 7, Tailwind 4, shadcn/ui, TanStack Query and Table, React Router 7, and Recharts. Pages live in `frontend/src/pages/`, mirroring the dashboard nav — `explorer/`, `visualize/`, plus top-level pages. + +It talks to the backend over the same public REST API documented in [API reference](api-reference.md); there is no privileged channel. + +## Next + +- [Data model](data-model.md) — the tables these slices own. +- [Extending ForecastLabAI](extending.md) — adding to this structure safely. diff --git a/docs/manual/integrator/data-model.md b/docs/manual/integrator/data-model.md new file mode 100644 index 00000000..998cd2e8 --- /dev/null +++ b/docs/manual/integrator/data-model.md @@ -0,0 +1,120 @@ +# Data model + +The twenty-three tables, grouped by the slice that owns them, and how they relate. + +**Purpose:** read or query the database without reverse-engineering it from ORM classes. +**Intended reader:** integrators querying directly, or adding a schema change. + +## What you'll accomplish + +A map of what is stored where, and which tables feed which features. + +## Ownership + +Every table belongs to exactly one slice, defined in that slice's `models.py`. Nothing is shared-write across slices — a slice that needs another's data goes through its service, not its tables. + +## The retail core — `data_platform` (10 tables) + +The warehouse the whole system forecasts over. + +| Table | Holds | +|---|---| +| `store` | Retail locations — region, store type. | +| `product` | SKUs — category, brand. | +| `calendar` | The date dimension: day-of-week, month, holiday flags. | +| `sales_daily` | **The fact table.** Daily units and revenue per (store, product). | +| `price_history` | Price level over time. | +| `promotion` | Promotional periods and their kind. | +| `inventory_snapshot_daily` | Daily on-hand stock per (store, product). | +| `replenishment_event` | Inbound stock arrivals. | +| `sales_returns` | Return events. | +| `exogenous_signal` | External drivers — weather, macro indicators. | + +`sales_daily` at the **(store, product, date)** grain is the spine. Everything else either describes an entity (`store`, `product`, `calendar`) or explains a movement in it. + +### These tables are the feature packs + +The last six map directly onto V2 feature packs, which is the practical reason to care about them: + +| Table | Feeds pack | +|---|---| +| `price_history`, `promotion` | `price_promo` | +| `inventory_snapshot_daily` | `inventory` | +| `replenishment_event` | `replenishment` | +| `sales_returns` | `returns` | +| `exogenous_signal` | `exogenous_weather`, `exogenous_macro` | + +**A pack whose table was never populated contributes nothing.** Enabling `inventory` against a dataset seeded without inventory dynamics adds empty columns, not signal. This is why the five sidecar packs are off by default — see [Forecasting](../analyst/forecasting.md). + +## Modelling and orchestration + +| Table | Slice | Holds | +|---|---|---| +| `job` | `jobs` | One async unit of work: type, params, status, result JSON, error, linked run. | +| `batch_job` | `batch` | A matrix submission. | +| `batch_job_item` | `batch` | One expanded child of a batch. | +| `model_run` | `registry` | **One training execution**: config, metrics, artifact location, status. | +| `deployment_alias` | `registry` | A movable pointer to one successful run. | +| `model_selection_run` | `model_selection` | One champion-selector comparison. | +| `model_selection_candidate` | `model_selection` | One candidate within a comparison, with its backtest result. | +| `scenario_plan` | `scenarios` | A saved what-if plan and its assumptions. | +| `forecast_explanation` | `explainability` | Stored forecast explanations. | +| `showcase_workspace` | `demo` | Saved showcase workspace state. | + +### `model_run` is the centre of gravity + +Nearly every question about "what happened" resolves to a `model_run` row: what was trained, at what grain, over which data window, with which configuration and seed, scoring what, backed by which artifact and checksum. + +Two facts about it are easy to get wrong: + +- **`model_family` is not a column.** It is computed from `model_type` at response time by `app/shared/model_taxonomy.py`. Querying the table for a family means filtering on the model types that map to it. +- **`feature_frame_version` may be absent on older rows.** Runs predating the field default to V1 for comparability purposes. + +A run's lifecycle is `pending → running → success`, or `failed`. An alias may point **only** at a successful run. + +## Conversational + +| Table | Slice | Holds | +|---|---|---| +| `document_source` | `rag` | An indexed document: path and content hash. | +| `document_chunk` | `rag` | A chunk with its **pgvector** embedding. | +| `agent_session` | `agents` | One conversation with its message history and state. | + +`document_source` carries the content hash that makes indexing idempotent: same path, same hash → nothing to do. + +`document_chunk.embedding` is a **fixed-width** pgvector column sized by `rag_embedding_dimension`. Changing embedding model to one with a different width is therefore a **migration plus a full re-index**, not a settings change. This is the single most common RAG configuration mistake — see [Troubleshooting](../troubleshooting.md). + +`agent_session` is where the `awaiting_approval` state lives when the human-in-the-loop gate pauses an agent. + +## Configuration + +| Table | Slice | Holds | +|---|---|---| +| `app_config` | `config` | Persisted runtime setting overrides. | + +This is what makes **Admin → AI models** changes survive a restart: overrides are written here and re-applied onto the `Settings` singleton at startup by `apply_overrides_on_startup`. It is the only table that can change application behavior without a redeploy, and it holds provider API keys — which is why `GET /config/ai` always masks them. + +## Migrations + +Schema lives in `alembic/versions/`. Three rules: + +1. **Every schema change ships a migration.** No implicit table creation. +2. **Migrations are forward-only once merged.** Editing a merged migration is forbidden — add a new one. +3. **Run `uv run alembic upgrade head` after every pull.** A missed migration surfaces as `relation "…" does not exist`. + +The database is PostgreSQL 16 with the **pgvector** extension (`pgvector/pgvector:pg16`), which the RAG tables require. + +## Querying directly + +```bash +psql postgresql://forecastlab:forecastlab@localhost:5433/forecastlab +``` + +Read-only exploration is fine. Writing directly is not: it bypasses Pydantic validation, the service-layer invariants, and the registry's artifact bookkeeping. A hand-written `model_run` row with no verifiable artifact will pass a metrics display and fail the promotion gate. + +Note also that all application access is **async** SQLAlchemy with parameter binding. Building SQL by string concatenation is forbidden repository-wide. + +## Next + +- [Artifacts and the registry](artifacts-and-registry.md) — what lives on disk beside these rows. +- [Extending ForecastLabAI](extending.md) — adding a table safely. diff --git a/docs/manual/integrator/extending.md b/docs/manual/integrator/extending.md new file mode 100644 index 00000000..2d0c06f2 --- /dev/null +++ b/docs/manual/integrator/extending.md @@ -0,0 +1,135 @@ +# Extending ForecastLabAI + +What to change, how to change it, and the short list of things that must not change. + +**Purpose:** add capability without breaking the guarantees the rest of the manual describes. +**Intended reader:** integrators contributing code. + +## What you'll accomplish + +A change that fits the architecture, ships with its tests and migration, and passes the gates on the first try. + +## Before anything: what must not change + +These are not style preferences. Each one, if broken, silently invalidates something the system claims. + +| Invariant | Why it exists | +|---|---| +| **Never weaken `app/features/featuresets/tests/test_leakage.py`.** | It *is* the leakage specification. Weakening it makes every accuracy number unfalsifiable. | +| **Never edit a merged Alembic migration.** | Migrations are forward-only. Editing one diverges every database that already applied it. Add a new migration. | +| **Never import one feature slice from another.** | The one-way import graph is what keeps 19 slices tractable. Shared types go in `app/shared/`. | +| **Never read `os.environ` in feature code.** | `get_settings()` is what makes the [configuration reference](../configuration.md) complete rather than approximate. | +| **Never build SQL by string concatenation.** | Parameter binding only. | +| **Never add a managed-cloud SDK to `app/`.** | It violates the single-host vision that makes `docker compose up` sufficient. | +| **Never widen an agent's mutation surface without adding the tool to `agent_require_approval`.** | That list is the human-in-the-loop boundary. | +| **Never skip Ruff, mypy `--strict`, or pyright `--strict`.** | All three gate merge. | +| **Never mock the database in integration tests.** | They must run against real Compose Postgres or they prove nothing. | +| **Never add an AI co-author or "Generated with" commit trailer.** | A hook enforces this. | + +The full list lives in [AGENTS.md](../../../AGENTS.md); this is the subset an extension is most likely to hit. + +## Adding a forecasting model + +The model taxonomy is deliberately small to change: + +1. **Implement the model** in `app/features/forecasting/`, following the existing model classes. +2. **Add the `model_type` literal** to the `ModelType` union in `app/features/forecasting/models.py`. +3. **Add the mapping** to `_MODEL_FAMILY_MAP` in `app/shared/model_taxonomy.py`. A drift-lock test asserts the map covers every known model type — miss this and it fails. +4. **Gate it if it needs an optional dependency**: add a `forecast_enable_` setting and a `pyproject.toml` extra, following `lightgbm` and `xgboost`. Pure-scikit-learn models need only the flag, like `random_forest`. +5. **Decide the feature frame.** A model consuming features belongs to `tree` or `additive` and gains V2 support; a target-only model is a `baseline` and must reject V2. +6. **Ship tests.** + +**Forgetting step 3 does not crash.** `model_family_for` classifies unknown types as `baseline` and logs a warning — forward-compatible by design. Your model will simply appear as a baseline in the dashboard, get the wrong badge, and be excluded from feature-importance routing. The drift-lock test exists precisely because this failure is quiet. + +`KNOWN_MODEL_TYPES` is derived from the same map, so cross-slice validation updates itself. + +## Adding a feature pack + +1. Add the member to `FeatureGroup` in `app/shared/feature_frames/contract_v2.py`. +2. Add it to `_GROUP_ORDER` — the manifest emits columns in exactly this order. +3. Add its column manifest to `_GROUP_COLUMNS`. +4. Decide default membership. `DEFAULT_V2_GROUPS` holds six; sidecar packs reading tables a small seeded database may not populate stay **off**. +5. Add a safety class if it reads a column a production pipeline must supply — that becomes the `Requires supplied data` chip. +6. **Prove time-safety.** A pack reading a new table is exactly where leakage enters. + +Remember the semantics: a disabled group's columns are **omitted entirely**, not NaN-filled. A NaN inside an *enabled* group means "source data unknown for this day", which the tree models handle natively. + +## Adding a slice + +Only when a genuinely new domain appears — not for a variation on an existing one. + +``` +app/features// +├─ models.py Mapped[] + mapped_column() +├─ schemas.py Pydantic v2 +├─ service.py business logic +├─ routes.py APIRouter with a prefix +└─ tests/ +``` + +Then wire the router in `app/main.py`, ship an Alembic migration for any tables, and keep imports one-way. + +If your new slice needs a type from an existing one, that type belongs in `app/shared/` — see the `ModelFamily` history in [Code architecture](code-architecture.md#what-the-rule-actually-prevents--a-real-example). + +## Adding an endpoint + +- Validate with Pydantic v2 at the boundary. +- Raise domain exceptions from `app/core/exceptions.py`; never a bare `HTTPException` with a raw string. +- Choose the error type deliberately: `validation` (the input is wrong) versus `unprocessable-entity` (the state forbids it). Clients branch on this — see [API reference](api-reference.md#the-distinction-that-matters-most). +- Ship a route test covering the 2xx path **and at least one error path**. + +## Adding an agent tool + +Read-only tools run immediately. **A tool that mutates state must be added to `agent_require_approval`** — otherwise the agent can perform that mutation with nobody in the loop. + +That is why `save_scenario` joined `create_alias` and `archive_run` when the experiment agent gained scenario persistence: the capability and its gate landed together. + +## Changing configuration + +Add the field to `Settings` in `app/core/config.py`. It becomes settable by the upper-case environment variable automatically. + +Then: document it in the [configuration reference](../configuration.md), add it to `.env.example` if deployments will commonly change it, and note whether it needs a restart. Almost everything does — `get_settings()` is cached. The AI-model settings are the exception, going through the `app_config` override mechanism. + +## Schema changes + +```bash +uv run alembic revision --autogenerate -m "description" +# review the generated migration — autogenerate is a draft, not an answer +uv run alembic upgrade head +``` + +Review before committing: autogenerate misses server defaults, index intent, and data migrations. Once merged, it is immutable. + +## Tests + +- Every new module, public function, endpoint, ORM model, and migration ships with a test. +- Every bug fix ships a **regression test that would have caught it**. +- Unit tests mock external services — OpenAI, Anthropic, Ollama. +- Integration tests are marked `@pytest.mark.integration` and run against **real** Compose Postgres. + +```bash +uv run pytest -v -m "not integration" # no DB needed +uv run pytest -v -m integration # needs docker compose up +``` + +## Before you commit + +```bash +uv run ruff check . && uv run ruff format --check . +uv run mypy app/ && uv run pyright app/ +uv run pytest -v -m "not integration" +``` + +Frontend work adds `cd frontend && pnpm tsc --noEmit && pnpm lint && pnpm test --run`. + +Details in [CI and quality gates](ci-and-quality-gates.md). + +## Stop and ask before + +- Cutting `dev` → `main`, or pushing any tag — release-please owns tagging. +- Bumping pydantic-ai, FastAPI, or SQLAlchemy major versions. +- Widening an agent's mutation surface. + +## Next + +- [CI and quality gates](ci-and-quality-gates.md) — what runs, and the branch and commit conventions. diff --git a/docs/manual/operator/concepts.md b/docs/manual/operator/concepts.md new file mode 100644 index 00000000..d8caa73c --- /dev/null +++ b/docs/manual/operator/concepts.md @@ -0,0 +1,72 @@ +# What ForecastLabAI is + +The system in one chapter: what it does, the vocabulary it uses, and — just as important — what it deliberately is not. + +**Purpose:** enough shared understanding that every later chapter reads as obvious. +**Intended reader:** everyone, before anything else. No installation required to read this. + +## What you'll accomplish + +You will be able to say what problem this system solves, name the stages of its lifecycle, and explain why a measurement it produces is trustworthy as engineering evidence but not as a claim about real retail demand. + +## The problem + +A retailer asks a narrow, repeating question: **how many units of this product will this store sell over the next N days?** Answer it too low and you stock out; too high and you tie up cash in inventory that ages. + +ForecastLabAI answers that question at the grain of one **(store, product) pair**, one day at a time, and — more importantly — it shows its work: how the answer was produced, how accurate it has been historically, and who approved putting it into service. + +## The lifecycle + +Nine stages, each a working vertical slice rather than a specification: + +1. **Data platform** — the retail tables: stores, products, calendar, daily sales, prices, promotions, inventory, replenishment, returns, exogenous signals. +2. **Ingest** — loading sales through a batch API that is safe to re-run. +3. **Feature engineering** — turning raw history into model-ready columns *without leaking the future*. +4. **Forecasting** — training one of eleven model types across three families. +5. **Backtesting** — replaying history to measure how accurate a model would have been. +6. **Model registry** — recording every run with its config, metrics, and a checksummed artifact. +7. **RAG knowledge base** — semantic search over indexed documentation. +8. **Agentic layer** — chat agents that can answer questions and run experiments, behind a human approval gate. +9. **Dashboard** — a React app that surfaces all of the above. + +The stages are ordered but not rigid: you can backtest without promoting, explore without forecasting, and use the dashboard without touching a terminal. + +## Five ideas that explain most decisions + +**Leakage is the enemy.** A feature that peeks at the future makes a model look brilliant and be useless. Every feature is built with `shift(lag)` and `shift(1).rolling()` patterns so a future value structurally cannot reach the model, and `app/features/featuresets/tests/test_leakage.py` is treated as the specification — weakening it is forbidden. When a metric looks too good, leakage is the first suspect. + +**Baselines are not filler.** Five deliberately simple forecasters ship as first-class models. "Predict last week's same weekday" is often hard to beat, and a machine-learning model that cannot beat it is not worth its complexity. Without the baseline in the comparison you would never learn that. + +**A measurement is only as good as its provenance.** A run records its configuration, its data window, its seed, and a SHA-256 of its artifact. That is what lets two runs be compared honestly, and what lets the registry refuse to promote a model whose artifact no longer verifies. + +**Recommendation is not authority.** The system ranks candidates and names a winner. It does not promote it. A person approves, with their name recorded, and overriding the recommendation requires an explicit acknowledgement. Automation proposes; a human disposes. + +**Agents are bounded.** The chat agents can read freely and act only within a gate: every mutating tool pauses and waits for approval. Sessions are capped in tokens, tool calls, and wall-clock time. + +## What this is not + +- **Not a multi-tenant SaaS.** Single host, no tenancy model, no auth boundary between users. +- **Not real-time.** Daily grain, batch jobs, asynchronous work. Nothing streams. +- **Not cloud-dependent.** No managed-cloud SDK is permitted in the core path; that rule is in [AGENTS.md](../../../AGENTS.md) and it is what keeps `docker compose up` sufficient. +- **Not trained on real retail data.** The dataset comes from a synthetic generator. See below — this is the most important caveat in the manual. +- **Not an inventory optimiser.** The safety-stock figure is a labelled heuristic over demand variability, not a full optimisation, and it never influences model ranking. + +## The honesty caveat about data + +Everything ForecastLabAI measures, it measures on data it generated itself, from a seed, via the synthetic seeder known as **The Forge**. + +That has two consequences, and this manual holds both at once: + +- The measurements are **real and reproducible**. Same seed, same scenario, same configuration produces the same dataset and comparable runs. The metrics are correct measurements of model behavior, the leakage controls genuinely hold, and the artifact checksums genuinely verify. +- The measurements say **nothing about real-world retail demand**. A model that wins here won on patterns the generator put there. Carrying a conclusion from this system to a real business would require real data and a fresh validation of every claim. + +This manual never blurs that line. Where a number appears, it is a configured default or a fixed constant you can check; runtime figures — durations, accuracy scores, which model wins — depend on your data, seed, and hardware, so the manual teaches you to *read* them rather than asserting values it cannot reproduce on your machine. + +## The shape of the code + +Nineteen **vertical slices** under `app/features/`, each owning its models, schemas, service, routes, and tests. A slice may not import another slice — shared code goes through `app/core/` or `app/shared/`. That single rule is why the system stays legible at this size, and it is covered in [Code architecture](../integrator/code-architecture.md). + +## Next + +- [Installation](installation.md) — prerequisites and first run. +- [Glossary](../glossary.md) — every term above, defined precisely. diff --git a/docs/manual/operator/installation.md b/docs/manual/operator/installation.md new file mode 100644 index 00000000..3fe0ace2 --- /dev/null +++ b/docs/manual/operator/installation.md @@ -0,0 +1,152 @@ +# Installation + +From an empty checkout to a backend that answers `/health` and a dashboard that loads. + +**Purpose:** get the stack running, with each step's pass condition stated. +**Intended reader:** operators installing for the first time. + +## What you'll accomplish + +A migrated PostgreSQL database, a backend serving on `:8123`, and a dashboard on `:5173`. The database will still be **empty** — filling it is [Seeding data](seeding-data.md), and the fastest route to a working system is [Quickstart](quickstart.md). + +## Prerequisites + +| Requirement | Why | +|---|---| +| **Docker + Docker Compose v2** | PostgreSQL 16 with the pgvector extension. | +| **Python 3.12+** with [`uv`](https://docs.astral.sh/uv/) | Backend runtime and dependency management. | +| **Node.js 20+** with `pnpm` (via `corepack`) | The dashboard. Skip if you only want the API. | + +An LLM API key is **optional** — see [Optional: AI features](#optional-ai-features). + +## 1 · Configure the environment + +```bash +cp .env.example .env +``` + +`.env` is never committed; only `.env.example` is tracked. The defaults work as-is for a local install — you only need to edit it for the optional AI features. + +Every variable is documented in the [configuration reference](../configuration.md). Note that `.env.example` ships the commonly-changed subset, not the full surface. + +## 2 · Start PostgreSQL + pgvector + +```bash +docker compose up -d +``` + +**Pass condition:** `docker compose ps` shows the Postgres service healthy, publishing host port **5433**. + +Port 5433 is deliberate — it avoids colliding with a Postgres you may already run on 5432. The container still listens on 5432 internally; 5433 is only the host-side publication. + +## 3 · Install backend dependencies + +```bash +uv sync --extra dev +``` + +Two forecasting models are opt-in extras. Add them now if you want them: + +```bash +uv sync --extra dev --extra ml-lightgbm # then set FORECAST_ENABLE_LIGHTGBM=true +uv sync --extra dev --extra ml-xgboost # then set FORECAST_ENABLE_XGBOOST=true +``` + +Installing the extra is only half — each model also needs its `forecast_enable_*` flag set to `true`. The flags are permission gates, not installation checks, so setting one without the library fails later at fit or unpickle time rather than at startup. `random_forest` is the exception: set `FORECAST_ENABLE_RANDOM_FOREST=true` and nothing else, since it is pure scikit-learn. + +## 4 · Apply database migrations + +```bash +uv run alembic upgrade head +``` + +**Pass condition:** `uv run alembic current` reports a revision. Migrations are forward-only once merged — after any `git pull`, run this again. + +## 5 · Verify database connectivity + +```bash +uv run python scripts/check_db.py +``` + +This confirms the application can reach *and authenticate to* the database — a stricter check than the container being healthy. + +## 6 · Start the backend + +```bash +uv run uvicorn app.main:app --reload --port 8123 +``` + +**Pass condition:** + +```bash +curl http://localhost:8123/health +# {"status":"ok"} +``` + +The interactive OpenAPI contract is now at **http://localhost:8123/docs**. That schema is generated from the code and is the authoritative API reference; this manual explains it rather than restating it. + +## 7 · Start the dashboard + +In a second terminal: + +```bash +cd frontend +corepack enable pnpm +pnpm install +pnpm dev +``` + +**Pass condition:** http://localhost:5173 loads. With an empty database the KPI cards read zero — that is correct, not a failure. + +The dashboard reads one variable of its own, `VITE_API_BASE_URL` (default `http://localhost:8123`), from `frontend/.env`. It is a **build-time** variable: changing it requires restarting `pnpm dev`, not just a hot reload. + +## Ports + +| Service | URL | +|---|---| +| Dashboard | http://localhost:5173 | +| Backend API | http://localhost:8123 | +| API docs (OpenAPI) | http://localhost:8123/docs | +| PostgreSQL | localhost:5433 | + +## Optional: AI features + +The chat agents and OpenAI-backed embeddings need a key in `.env`: + +```bash +OPENAI_API_KEY=sk-… +# and/or +ANTHROPIC_API_KEY=sk-ant-… +``` + +Without a key, **forecasting, backtesting, the registry, the Explorer, and every analytical page still work.** Only `/chat` and OpenAI embeddings are unavailable. + +To avoid external services entirely, run embeddings and the agent locally through Ollama — set `RAG_EMBEDDING_PROVIDER=ollama` and an `ollama:` agent model. Mind `rag_embedding_dimension`: it must match the model's output width (1536 for OpenAI `text-embedding-3-small`, 768 for `nomic-embed-text`). It is a fixed-width column, so changing it is a migration, not a settings edit. + +These settings are also editable at runtime with no restart from **`/admin` → AI Models** — see [Runtime-editable settings](../configuration.md#runtime-editable-settings-no-restart). + +## Everything in containers instead + +If you would rather not install Python and Node locally: + +```bash +make docker-up # full stack in containers +make docker-up-gpu # same, plus Ollama on GPU +``` + +Container mode changes how the backend reaches the database (`postgres:5432`, not `localhost:5433`). See [Running the stack](running-the-stack.md). + +## Verifying the whole install + +The honest end-to-end check is the demo pipeline, which exercises seed → features → train → backtest → register → alias → agent: + +```bash +make demo +``` + +`make demo` requires the backend to **already be serving** on `:8123` — it drives the running API rather than starting one. See [Quickstart](quickstart.md). + +## Next + +- [Quickstart](quickstart.md) — get to a working system with trained models. +- [Troubleshooting](../troubleshooting.md) — if a pass condition above did not hold. diff --git a/docs/manual/operator/operations.md b/docs/manual/operator/operations.md new file mode 100644 index 00000000..87e6e3d4 --- /dev/null +++ b/docs/manual/operator/operations.md @@ -0,0 +1,119 @@ +# Operations + +Running the system past the first demo: work queues, batches, artifacts, health, and the upkeep that keeps results trustworthy. + +**Purpose:** the day-two concerns — what accumulates, what can wedge, and what to watch. +**Intended reader:** operators maintaining a working install. + +## What you'll accomplish + +An understanding of how asynchronous work flows through the system, where its outputs land on disk, and how to read the operational surface at `/ops`. + +## Jobs: the unit of asynchronous work + +Training, prediction, and backtesting all run as **jobs** rather than blocking HTTP calls. + +| Endpoint | Purpose | +|---|---| +| `POST /jobs` | Submit a `train`, `predict`, or `backtest` job; returns a `job_id`. | +| `GET /jobs` | List jobs with filters and sorting. | +| `GET /jobs/{job_id}` | Status and result JSON. | +| `DELETE /jobs/{job_id}` | Cancel a pending job. | + +A job carries its parameters, its result, any error detail, and a link to the model run it produced. **Explorer → Jobs** is the same data in the dashboard, with live status polling and a cancel action. + +`DELETE` cancels a **pending** job cleanly. A job already executing a model fit is a different matter — see [Cancellation and drain](#cancellation-and-drain). + +Job records are retained for `jobs_retention_days` (default 30). + +## Batches: many jobs as one submission + +A **batch** expands a matrix — (store, product) pairs × model configurations — into many child items and runs them with bounded concurrency. + +| Endpoint | Purpose | +|---|---| +| `POST /batch` | Submit a batch. | +| `GET /batch` | List batches. | +| `GET /batch/{batch_id}` | Batch status and per-item results. | +| `DELETE /batch/{batch_id}` | Cancel a batch (drains in-flight children). | + +Two limits shape behavior, and both fail loudly rather than silently degrading: + +**Scope.** Expanded scope is capped by `batch_max_scope_expansion` (default 1000). A batch that would expand past it is **rejected at submission**, not queued and truncated. Narrow the pair list or the model matrix. + +**Concurrency.** Effective parallelism is `min(batch_job.max_parallel, batch_global_max_parallel)`. The global default is `4`, sized for the Compose Postgres pool (`pool_size=5`, `max_overflow=10`). Raising it without raising the pool surfaces as connection-pool exhaustion under load, not as faster throughput. + +The dashboard equivalent is **Visualize → Batch Runner**, which offers five prefilled sweep presets — see [Backtesting](../analyst/backtesting.md). + +## Cancellation and drain + +**A scikit-learn or LightGBM fit cannot be cancelled mid-call.** This is a property of the libraries, not a gap in the system, and it shapes the cancellation contract. + +When you cancel a batch or a champion-selection run, the API stops scheduling new work and then *waits* for in-flight fits to finish. That wait is bounded: + +- `batch_cancel_drain_timeout_seconds` (default 30) +- `model_selection_cancel_drain_timeout_seconds` (default 30) + +Exceeding the timeout returns an RFC 7807 **504**. That response means "the drain did not complete within the window" — **not** "cancellation failed". The batch is still cancelling; a long fit is simply still running. Re-check status rather than re-issuing the cancel. + +## The operational surface: `/ops` + +Three read-only endpoints back the Control Center page: + +| Endpoint | What it answers | +|---|---| +| `GET /ops/summary` | Overall operational state. | +| `GET /ops/retraining-candidates` | Which models look due for a refresh. | +| `GET /ops/model-health` | Health of registered models and their aliases. | + +The page surfaces **stale aliases** as a dedicated card with a reason chip per row — `newer success run`, `artifact not verified`, `run not success`, or `V mismatch`. These are the routine signals that a promoted model needs attention. Their meanings and the promotion gate are covered in [Champion selector](../analyst/champion-selector.md). + +Of the four, **`artifact not verified` is the one to treat as urgent**: it means the file backing a promoted alias no longer matches its recorded SHA-256. + +## What accumulates on disk + +Four artifact roots, all configurable: + +| Setting | Default | Holds | +|---|---|---| +| `forecast_model_artifacts_dir` | `./artifacts/models` | Fitted model artifacts. | +| `backtest_results_dir` | `./artifacts/backtests` | Backtest result files. | +| `registry_artifact_root` | `./artifacts/registry` | Registry-tracked artifacts. | +| `showcase_export_root` | `./artifacts/showcase` | Workspace export bundles with manifests and checksums. | + +In container mode these live in the `forecastlab_artifacts` named volume, which survives `make docker-down`. + +**Artifacts are what make a run verifiable.** Deleting a model artifact does not delete its registry row — it produces a run whose metrics still display but whose artifact verification now fails, and whose feature-importance endpoint returns `422`. If you are reclaiming space, archive runs through the registry rather than deleting files underneath it. + +## Model runs and duplicates + +`registry_duplicate_policy` (default `detect`) decides what happens when a run duplicates an existing one: + +- `detect` — flag the duplicate but allow it. +- `deny` — reject it. +- `allow` — record it without comment. + +`detect` is the useful default: it surfaces accidental repeat work without blocking a deliberate re-run. + +## Logs + +Structured logging via `structlog`. Two settings control it: `log_level` (default `INFO`) and `log_format` — `json` for machine-readable output, `console` for readable local development. + +Every request carries a request ID via `RequestIdMiddleware`, which is the join key between a client-side failure and the server-side log lines for that request. + +**Log key names, never key values.** That rule is in [AGENTS.md](../../../AGENTS.md); secrets must never reach a log line, and `GET /config/ai` masks API keys for the same reason. + +## Routine upkeep + +**After every `git pull`:** `uv run alembic upgrade head`. Migrations are forward-only, and a missed one surfaces as `relation "…" does not exist`. + +**Before trusting a promoted model:** check `GET /ops/model-health` and confirm no alias is flagged `artifact not verified`. + +**When comparing models:** hold the dataset fixed. Changing seed or scenario *and* model at once measures nothing. See [Seeding data](seeding-data.md). + +**When results look surprisingly good:** suspect leakage first, and run `app/features/featuresets/tests/test_leakage.py`. Then remember the data is synthetic — see [Backtesting](../analyst/backtesting.md). + +## Next + +- [Dashboard tour](../analyst/dashboard-tour.md) — the operational pages in the UI. +- [Artifacts and the registry](../integrator/artifacts-and-registry.md) — the integrity contract in detail. diff --git a/docs/manual/operator/quickstart.md b/docs/manual/operator/quickstart.md new file mode 100644 index 00000000..90ed8f70 --- /dev/null +++ b/docs/manual/operator/quickstart.md @@ -0,0 +1,106 @@ +# Quickstart + +The shortest honest path from a migrated-but-empty install to a system with data, trained models, a backtested winner, and a promoted alias. + +**Purpose:** exercise the whole lifecycle in one command, and know what each step proved. +**Intended reader:** operators who finished [Installation](installation.md), and reviewers who want to see the system work. + +## What you'll accomplish + +A seeded dataset, three trained models, three backtests, a registered winner with a verified artifact, an alias pointing at it, and an agent round-trip — with a printed verdict. + +## Preconditions + +`make demo` is a **black-box HTTP consumer**: it drives the published API rather than starting anything. Two things must already be true: + +1. Postgres is reachable on `:5433` (the target starts it for you). +2. **The backend is already serving on `http://localhost:8123`.** + +```bash +uv run uvicorn app.main:app --reload --port 8123 # in its own terminal +``` + +If the API is unreachable, the demo exits `2` — a precondition failure, distinct from a step failure. + +## Run it + +```bash +make demo +``` + +The target runs `docker compose up -d`, applies migrations, and then drives the pipeline with seed 42. + +## What it does + +``` +precheck → (reset) → seed → status → features + → train ×3 (parallel) → backtest ×3 (sequential) + → register-winner → verify → agent → cleanup +``` + +Each stage proves something specific: + +| Stage | What a pass proves | +|---|---| +| **precheck** | The API and database are reachable — the environment is real. | +| **seed** | The Forge can generate a reproducible dataset from seed 42. | +| **features** | Time-safe feature computation runs over the seeded history. | +| **train ×3** | Three model types fit and produce artifacts, concurrently. | +| **backtest ×3** | Time-series cross-validation scores all three on identical folds. | +| **register-winner** | The registry records the ranked winner with its metrics. | +| **verify** | The winner's artifact passes SHA-256 verification. | +| **agent** | The agent layer answers through the live stack. | + +The comparison is meaningful precisely because the three models are backtested on the **same folds** — that is what makes "winner" a claim rather than an impression. + +## Reading the outcome + +A green run ends with a summary line naming the run count, the winning model, the alias, and wall-clock time — for example: + +``` +runs=3 winner=seasonal_naive alias=demo-production wall_clock=87s +``` + +**Which model wins is not fixed**, and a baseline winning is a legitimate result, not a bug. On synthetic data with a short history, `seasonal_naive` frequently beats feature-aware models — that is exactly why the baselines are in the comparison. + +Exit codes: + +| Code | Meaning | +|---|---| +| `0` | Green (possibly with a soft warning on the wall-clock budget). | +| `1` | One or more pipeline steps failed — a real observation about the system. | +| `2` | Precondition failure: the API or database was unreachable. Nothing was measured. | + +Do not read a `2` as evidence about the models — nothing ran. Only a `1` is a statement about the pipeline itself. + +## The same thing in the browser + +With the backend and dashboard both running, open **http://localhost:5173/showcase** and click **Run pipeline**. The identical flow streams into the page as one status card per step, each flipping to pass / fail / skip, with a summary banner naming the winner. + +Tick **Re-seed first** if the database is empty or stale. Only one pipeline may run at a time. + +This is the best route for a guided demo — no terminal, same evidence. + +## Variants + +```bash +make demo-quick # skip re-seeding — fast iteration on existing data +make demo-clean # DESTRUCTIVE: wipe the database first, then run +``` + +`make demo-clean` drops your seeded data. Use it when you want a guaranteed-clean measurement; avoid it if you have a dataset you care about. + +## Where to go from here + +You now have a working system. Depending on who you are: + +- **Explore what was built** → [Dashboard tour](../analyst/dashboard-tour.md). +- **Train your own model deliberately** → [Forecasting](../analyst/forecasting.md). +- **Understand the winner** → [Backtesting](../analyst/backtesting.md). +- **Do it properly, with promotion** → [Champion selector](../analyst/champion-selector.md). +- **Generate different data** → [Seeding data](seeding-data.md). +- **Call it from code** → [API reference](../integrator/api-reference.md). + +## Next + +- [Seeding data](seeding-data.md) — the eight scenario presets and what each stresses. diff --git a/docs/manual/operator/running-the-stack.md b/docs/manual/operator/running-the-stack.md new file mode 100644 index 00000000..a20d07f7 --- /dev/null +++ b/docs/manual/operator/running-the-stack.md @@ -0,0 +1,105 @@ +# Running the stack + +The two ways to run ForecastLabAI — host mode and container mode — how they differ, and when the difference bites. + +**Purpose:** choose a mode deliberately and know what changes when you switch. +**Intended reader:** operators running the system beyond a first install. + +## What you'll accomplish + +A running stack in whichever mode suits you, and a clear model of which hostname reaches which service from where. + +## The two modes + +**Host mode** — Postgres in Docker; backend and frontend run directly on your machine. Best for development: `--reload` works, breakpoints work, edits are instant. + +**Container mode** — everything in Docker, including backend and frontend. Best for demos and for "does this work on a clean machine" checks. + +They differ in exactly one thing that matters: **network identity**. + +## The hostname rule + +| From | Reaches Postgres at | Reaches Ollama at | +|---|---|---| +| Your machine (host mode) | `localhost:5433` | `localhost:11434` | +| Inside a container | `postgres:5432` | `ollama:11434` | + +The Compose file publishes Postgres as `5433:5432` — host port 5433, container port 5432. Host port 5433 avoids colliding with a Postgres you may already run. + +The backend container sets `DATABASE_URL` and `OLLAMA_BASE_URL` in its own `environment:` block, which **overrides whatever `.env` holds**. This is why `.env` can keep the host-mode defaults: container mode does not read them. + +Nearly every "works from my terminal but not in the container" problem is this table. + +## Host mode + +```bash +docker compose up -d # Postgres only +uv run alembic upgrade head +uv run uvicorn app.main:app --reload --port 8123 # terminal 1 +cd frontend && pnpm dev # terminal 2 +``` + +## Container mode + +```bash +make docker-up # docker compose up -d --wait --wait-timeout 90 +``` + +`--wait` blocks until the healthchecks pass, so the command returning means the stack is genuinely ready — not merely started. All three services (postgres, backend, frontend) declare healthchecks. + +Stop with: + +```bash +make docker-down # stops and removes containers, KEEPS named volumes +``` + +Your data survives `docker-down` because it lives in named volumes: `forecastlab_pgdata` (the database), `forecastlab_artifacts` (model artifacts), and `forecastlab_ollama_models` (pulled Ollama models). Removing those volumes — `docker compose down -v` — is what actually destroys data. + +## The GPU profile + +Ollama is behind a Compose **profile**, so it does not start unless you ask for it: + +```bash +make docker-up-gpu +# docker compose -f docker-compose.yml -f docker-compose.gpu.yml --profile gpu up -d --wait --wait-timeout 120 +``` + +This adds an `ollama` service on `:11434` with GPU device reservations from the overlay file. Two things to understand: + +- **The GPU is for the LLM only.** It accelerates local embeddings and local agent models. The forecasting models are scikit-learn, LightGBM, and XGBoost — **CPU only**, unaffected by this profile. Turning on GPU will not make training faster. +- **Verify host GPU support before invoking it.** The overlay requires a working NVIDIA container runtime; without it, the service fails to start rather than silently falling back to CPU. + +The longer wait timeout (120s vs 90s) exists because Ollama's first start is slower. + +## Ports + +| Service | Host port | Container port | Notes | +|---|---|---|---| +| PostgreSQL + pgvector | 5433 | 5432 | `pgvector/pgvector:pg16` | +| Backend API | 8123 | 8123 | | +| Dashboard | 5173 | 5173 | | +| Ollama | 11434 | 11434 | `gpu` profile only | + +## Checking health + +```bash +docker compose ps # container health +curl http://localhost:8123/health # {"status":"ok"} +uv run python scripts/check_db.py # app can reach AND authenticate +uv run alembic current # migrations applied +``` + +`/health` is a liveness probe. It answering `ok` means the backend is up and its database connection works — so a failure above that line is infrastructure, and a failure below it is application logic or your request. + +## Which settings need a restart + +Almost all of them: `get_settings()` is `@lru_cache`d, so the settings object is built once per process. + +The deliberate exception is the AI-model configuration — agent model, embedding provider/model/dimension, and provider API keys — which `/admin` → AI Models persists to the `app_config` table and applies live. Those overrides are re-applied at startup too, so they survive restarts. Everything else in the [configuration reference](../configuration.md) requires bouncing the backend. + +Settings that specifically call out a restart requirement: `batch_global_max_parallel`, `model_selection_global_max_parallel`, and the three `forecast_enable_*` model flags. + +## Next + +- [Operations](operations.md) — jobs, batches, artifacts, and routine upkeep. +- [Troubleshooting](../troubleshooting.md) — when a mode switch breaks something. diff --git a/docs/manual/operator/seeding-data.md b/docs/manual/operator/seeding-data.md new file mode 100644 index 00000000..2592affa --- /dev/null +++ b/docs/manual/operator/seeding-data.md @@ -0,0 +1,116 @@ +# Seeding data + +The Forge: how ForecastLabAI generates the retail data it forecasts, and how to control what that data looks like. + +**Purpose:** produce a dataset that stresses the behavior you want to study, reproducibly. +**Intended reader:** operators preparing a demo or an experiment. + +## What you'll accomplish + +A populated dataset with realistic time-series structure — trend, weekly and monthly seasonality, noise, promotions, stockouts, product lifecycles — generated from a seed you control, so the same inputs always produce the same data. + +## The honest framing, first + +The Forge produces **synthetic data**. Nothing it generates is a real sales record. That is not a limitation to apologise for — it is what makes the system reproducible and shareable. But it bounds every conclusion drawn downstream: a model that wins on Forge data won on patterns the Forge created. See [What ForecastLabAI is](concepts.md#the-honesty-caveat-about-data). + +## Reproducibility + +Generation is seeded. **The same scenario plus the same seed produces the same dataset**, which is what lets two model runs be compared honestly. The default is `seeder_default_seed = 42`, alongside `seeder_default_stores = 10` and `seeder_default_products = 50`. + +Change the seed to get a different-but-equally-valid world; keep it fixed when you are comparing models, because changing data *and* model at once measures nothing. + +## The eight scenario presets + +Each preset tunes the generator toward a different difficulty: + +| Scenario | What it stresses | +|---|---| +| `retail_standard` | The general case — ordinary trend, seasonality, and noise. The sensible default. | +| `holiday_rush` | A Q4 demand surge with Black Friday and Christmas structure. | +| `high_variance` | Noisy, hard-to-predict demand. Punishes overconfident models. | +| `stockout_heavy` | Frequent stockouts — censored demand, where observed sales understate true demand. | +| `new_launches` | Many products with short histories and ramp-up curves (100 products, a 45-day ramp). Hard for lag-based features. | +| `sparse` | Intermittent demand with gaps of 2–10 days. Where sMAPE misbehaves and WAPE earns its keep. | +| `demo_minimal` | A small, fast dataset for smoke-testing the pipeline. | +| `showcase_rich` | Sized for the `/showcase` demo — large enough that a V2 `prophet_like` run gets full horizon-bucket coverage. | + +Two details worth knowing: + +- **`holiday_rush` is calendar-pinned.** Its holiday dates and Q4 seasonality model a specific 2024 window and are *not* re-anchored to today. Pass an explicit `start_date` / `end_date` to shift it. Every other window-anchored preset follows today's date. +- **`demo_minimal` and `showcase_rich` tune their noise deliberately** to avoid a degenerate case where WAPE evaluates to NaN because a fold's actual demand sums to zero. If you build a custom config with very low demand, expect to meet that trap. + +## Generating data + +### From the dashboard + +**Admin → Data seeding.** Choose a scenario, set the seed, generate. The same tab hosts append, verify, and clear. This is the recommended route — no terminal, and the destructive actions are behind confirmations. + +### From the API + +```bash +# What is currently loaded? +curl http://localhost:8123/seeder/status + +# What scenarios exist? +curl http://localhost:8123/seeder/scenarios + +# Generate +curl -X POST http://localhost:8123/seeder/generate \ + -H 'Content-Type: application/json' \ + -d '{"scenario": "retail_standard", "seed": 42}' +``` + +The full endpoint set: + +| Endpoint | Purpose | +|---|---| +| `GET /seeder/status` | Current dataset state. | +| `GET /seeder/scenarios` | Available presets. | +| `GET /seeder/channels` | Available sales channels. | +| `GET /seeder/exogenous` | Exogenous signal data. | +| `POST /seeder/generate` | Generate a dataset from a scenario. | +| `POST /seeder/append` | Extend an existing dataset. | +| `POST /seeder/verify` | Check dataset integrity. | +| `DELETE /seeder/data` | Clear generated data. | + +Scenario presets are a *starting point*: explicit parameters in the request override the preset's values. + +### As part of the demo + +`make demo` seeds with seed 42 before running the pipeline. `make demo-quick` skips seeding to iterate on existing data. See [Quickstart](quickstart.md). + +## What gets generated + +The Forge populates the data-platform tables — stores and products, a calendar, daily sales, and the signals that explain them: price history, promotions, inventory snapshots, replenishment events, returns, and exogenous signals. + +That last group matters for modelling: the V2 feature packs `price_promo`, `inventory`, `replenishment`, `returns`, `exogenous_weather`, and `exogenous_macro` read exactly these tables. **A pack whose underlying signal was not seeded contributes nothing.** If you want to study inventory-aware forecasting, seed a scenario that generates meaningful inventory dynamics — `stockout_heavy` is the obvious one. See [Forecasting](../analyst/forecasting.md). + +## Two guards against destroying real data + +| Setting | Default | Effect | +|---|---|---| +| `seeder_allow_production` | `false` | Blocks seeding entirely when `app_env` is `production`. | +| `seeder_require_confirm` | `true` | Requires explicit confirmation for destructive seeder operations. | + +Both default to the safe position. `DELETE /seeder/data` and `make demo-clean` are genuinely destructive — they remove generated data with no undo. + +## Appending versus regenerating + +**Append** extends the existing dataset forward, preserving history and the entities already present. Use it to lengthen a series or simulate the arrival of new days. + +**Regenerate** replaces. Use it when changing scenario or seed. + +Appending after changing the seed produces a dataset with a discontinuity at the join — occasionally useful for testing robustness, usually just confusing. Prefer one seed per dataset. + +## Verifying + +```bash +curl -X POST http://localhost:8123/seeder/verify +``` + +Checks internal consistency — that sales reference real stores and products, dates fall inside the generated window, and the signal tables line up with the fact table. Worth running after an append, or when a model behaves strangely for reasons you cannot explain. + +## Next + +- [Running the stack](running-the-stack.md) — host mode, container mode, and the GPU profile. +- [Forecasting](../analyst/forecasting.md) — turning this data into models. diff --git a/docs/manual/troubleshooting.md b/docs/manual/troubleshooting.md new file mode 100644 index 00000000..cf3eaa83 --- /dev/null +++ b/docs/manual/troubleshooting.md @@ -0,0 +1,176 @@ +# Troubleshooting + +Symptom → cause → fix, for the failures this system actually produces. + +**Purpose:** get you unstuck without reading source. +**Intended reader:** operators and analysts; integrators should also read the error envelope in [API reference](integrator/api-reference.md). + +## First, locate the layer + +Four things can be independently broken. Check them in this order — each depends on the ones above it. + +``` +Postgres (:5433) → migrations → backend API (:8123) → dashboard (:5173) +``` + +```bash +docker compose ps # is Postgres up and healthy? +uv run python scripts/check_db.py # can the app reach and authenticate to it? +uv run alembic current # are migrations applied? +curl http://localhost:8123/health # is the API up? → {"status":"ok"} +``` + +If `/health` answers `ok`, the backend and its database connection are both fine, and the problem is above that line — in the dashboard, or in the specific request you are making. + +## Startup and connectivity + +**The dashboard shows "Loading…" on every panel.** +The frontend cannot reach the backend. Confirm the API answers (`curl http://localhost:8123/health`), then check `frontend/.env` has `VITE_API_BASE_URL=http://localhost:8123`. `VITE_API_BASE_URL` is a **build-time** variable — after changing it, restart `pnpm dev`; a hot reload will not pick it up. + +**`connection refused` to the database.** +Postgres is not up, or is on a different port. The Compose service publishes host port **5433**, not the default 5432, so a `DATABASE_URL` pointing at 5432 will fail against a healthy container. Run `docker compose up -d` and confirm with `docker compose ps`. + +**The database works from the host but not from the backend container.** +Different network, different hostname. In Compose mode the backend reaches Postgres at `postgres:5432` (service DNS, container port), and the backend container's `environment:` block sets `DATABASE_URL` accordingly — it overrides `.env`. The `.env` value is the *host-mode* default. See [Running the stack](operator/running-the-stack.md). + +**`relation "…" does not exist`.** +Migrations were never applied, or a new migration landed after your last pull. Run `uv run alembic upgrade head`. + +**The backend starts but every write fails after a `git pull`.** +Same cause. Migrations are forward-only; pull then migrate. + +## Empty dashboard, no error + +**KPI cards all read zero and tables are empty.** +The database is migrated but has no data. A fresh install is empty by design — nothing seeds itself. Generate a dataset from **Admin → Data seeding**, or run `make demo`, or call `POST /seeder/generate`. See [Seeding data](operator/seeding-data.md). + +**The Knowledge page shows an empty corpus.** +No documents are indexed yet. Index one from **Admin → RAG Sources** or `POST /rag/index`. An empty corpus also means the `rag_assistant` agent has nothing to cite. + +**Demand Planner shows no SKUs.** +It rolls up **completed `predict` jobs**. With no successful prediction jobs there is nothing to display. Train a model and run a forecast first. + +## Model training and forecasting + +**`422` when training with `feature_groups` on V1.** +Feature packs belong to the V2 feature frame. A V1 training request carrying `feature_groups` is rejected. Either pick V2 or drop the packs. See [Forecasting](analyst/forecasting.md). + +**The V2 option is disabled for the model I picked.** +You picked a baseline. Baselines do not consume features, so V2 has no meaning for them — the UI disables the combination and explains it in a tooltip. Pick a tree or additive model. + +**`lightgbm` / `xgboost` is missing from the model list.** +Both are opt-in twice over: install the extra *and* set the flag. + +```bash +uv sync --extra dev --extra ml-lightgbm # then FORECAST_ENABLE_LIGHTGBM=true +uv sync --extra dev --extra ml-xgboost # then FORECAST_ENABLE_XGBOOST=true +``` + +`random_forest` needs only the flag (`FORECAST_ENABLE_RANDOM_FOREST=true`) — it is pure scikit-learn. All three flags require a backend restart. + +**A model trains, then unpickling its artifact fails later.** +The flag was set but the extra is not installed in the environment doing the loading. The `forecast_enable_*` flags are permission gates, not installation checks — they do not verify the library is present. Install the matching extra. + +**Champion selector refuses to forecast: "blocked".** +Expected, not a bug. A feature-aware model needs a **future** feature frame to predict forward, and the selector will not fabricate one. Use the [What-If Planner](analyst/demand-and-planning.md), which builds the forward frame from explicit assumptions. + +**The comparison is refused with `400` for a store/product pair.** +Too little history at that grain for a valid cross-validation split. The page flags the pair as unusable before submitting. Pick a pair with more history, or seed a longer date range. + +**Feature importance is unavailable for a run.** +Three distinct causes, distinguished by status code: +- **`400`** — the run is a baseline. Baselines have no learned importance vector; nothing is wrong. +- **`404`** — the run or job is not in the registry. +- **`422`** — the run has no artifact yet (`pending` / `running` / `failed`), the artifact file was deleted from disk, an optional `ml-*` extra is missing at unpickle time, or the estimator simply does not expose one. Note that `regression` uses scikit-learn's `HistGradientBoostingRegressor`, which **does not** expose `feature_importances_` — a `422` there is permanent, not transient. + +## Backtesting and metrics + +**No RMSE tile on an older backtest.** +RMSE was added later; backtest jobs recorded before it landed do not carry the key, and the UI omits the tile rather than showing a zero. Re-run the backtest to get it. + +**No per-horizon-bucket card.** +It renders only when the response carries `bucketed_aggregated_metrics`. Older jobs and some configurations do not produce it. + +**Metrics look implausibly good.** +Suspect leakage before celebrating. Everything in the feature path is built to prevent it and `app/features/featuresets/tests/test_leakage.py` locks the guarantee — but if you have added a feature, that test is the thing to run. Also remember the data is synthetic: patterns the generator put in are patterns a model can find. + +## Registry and promotion + +**Promotion rejected `422`.** +Four separate preconditions, each with its own message: +- the alias name must match `^[a-z0-9][a-z0-9\-_]*$`; +- `approved_by` must be present — promotion is never anonymous; +- a **non-recommended** model needs `acknowledge_non_recommended=true`; +- the model must be **trained** first. + +**The Promote button stays disabled and no checkbox releases it.** +The candidate's artifact failed SHA-256 verification. That is the one gate with **no operator override**, deliberately — a corrupt or missing artifact is not a judgement call. Re-train to produce a verifiable artifact. + +**An alias is flagged stale.** +Read the reason chip: `newer success run` (a better candidate exists), `artifact not verified` (integrity failure), `run not success` (target failed or was archived), or `V mismatch` (feature-frame drift). Only the last one is subtle — it means the alias's feature contract would silently change, which matters if a production pipeline feeds it. + +**Two runs will not compare.** +They must share a grain, have overlapping data windows, **and** share a `feature_frame_version`. The Compare page shows a champion-compatibility badge naming which condition failed. Runs predating the field default to V1. + +## Batches and cancellation + +**`DELETE /batch/{id}` returns `504`.** +The drain timed out. In-flight scikit-learn and LightGBM fits cannot be cancelled mid-call, so cancellation waits for them — bounded by `batch_cancel_drain_timeout_seconds` (default 30). The batch is still cancelling; the response only means it did not finish within the window. + +**A batch is slower than its `max_parallel` suggests.** +Effective parallelism is `min(batch_job.max_parallel, batch_global_max_parallel)`, and the global default is `4`. + +**Connection-pool exhaustion under load.** +`batch_global_max_parallel` was raised without raising the Postgres pool. The default of 4 is sized for the Compose pool (`pool_size=5`, `max_overflow=10`). + +**A batch is rejected before it starts.** +Expanded scope (pairs × model configs) exceeded `batch_max_scope_expansion` (default 1000). Narrow the scope. + +## RAG and agents + +**Indexing fails, or search returns nothing sensible after switching embedding models.** +`rag_embedding_dimension` must equal the width the model emits — OpenAI `text-embedding-3-small` is 1536, `nomic-embed-text` is 768. The vector column has a **fixed width**, so this is a schema mismatch, not a tuning problem: changing dimension requires a migration and re-indexing the corpus, not just a settings edit. + +**Retrieval returns nothing for an obviously relevant question.** +`rag_similarity_threshold` (default 0.7) is a floor — passages below it are dropped. Either the corpus lacks the content or the threshold is too strict for your embedding model. + +**Chat is unavailable but everything else works.** +The agents need an LLM API key (`OPENAI_API_KEY` or `ANTHROPIC_API_KEY`). Forecasting, backtesting, the registry, and the whole dashboard work without one. + +**Agent model rejected at startup.** +`agent_default_model` and `agent_fallback_model` are validated as `provider:model-name`. The three rejections: no colon, blank model name, or a **nested provider prefix** (`google-gla:google-gla:gemini-3-flash` — the error suggests the correction). Multi-colon Ollama tags like `ollama:llama3.1:8b` are valid. + +**The agent stops mid-task and waits.** +Working as designed. It hit a tool in `agent_require_approval` (`create_alias`, `archive_run`, `save_scenario` by default) and entered `awaiting_approval`. Approve or reject it in the chat, or via `POST /agents/sessions/{id}/approve`. Pending approvals expire after `agent_approval_timeout_minutes` (60). + +**The agent gives up partway through a longer task.** +Session bounds: `agent_max_tool_calls` (10), `agent_timeout_seconds` (120), `agent_max_tokens` (4096). The `/guide` page shows the live limits. + +**Ollama works on the host but not in Compose.** +`OLLAMA_BASE_URL` must be `http://ollama:11434` inside the Compose network, not `localhost`. GPU Compose mode injects this automatically. + +## Reading an API error + +Every endpoint returns RFC 7807 `application/problem+json` with a stable `type` URI under `/errors`. The code tells you which class of problem it is: + +| `type` | Typical status | Meaning | +|---|---|---| +| `/errors/validation` | 422 | The **input** failed validation. | +| `/errors/unprocessable-entity` | 422 | Input was well-formed but the **state** does not permit the action. | +| `/errors/bad-request` | 400 | The request does not apply to this resource at all. | +| `/errors/not-found` | 404 | No such resource. | +| `/errors/conflict` | 409 | Conflicts with existing state. | +| `/errors/gateway-timeout` | 504 | A drain or upstream wait timed out. | +| `/errors/embedding-auth` | — | The embedding provider rejected the credential. | +| `/errors/agent-fallback-exhausted` | — | Primary *and* fallback agent models both failed. | +| `/errors/database` | 500 | Database-level failure. | +| `/errors/internal` | 500 | Unhandled server error. | + +`validation` and `unprocessable-entity` are deliberately distinct: the first means *you sent something wrong*, the second means *what you sent cannot be done right now*. + +## Still stuck + +- [Configuration reference](configuration.md) — confirm the setting is what you think it is, and whether it needs a restart. +- [FAQ](faq.md) — several "is this broken?" questions answered as "no, here's why". +- `docs/_base/RUNBOOKS.md` — the deeper operational runbooks. +- The interactive API contract at `http://localhost:8123/docs` is always current; this manual is not generated from it. diff --git a/docs/user-guide/advanced-forecasting-guide.md b/docs/user-guide/advanced-forecasting-guide.md index 41a67ba6..5e28bbb7 100644 --- a/docs/user-guide/advanced-forecasting-guide.md +++ b/docs/user-guide/advanced-forecasting-guide.md @@ -1,179 +1,14 @@ # Advanced Forecasting Guide -This guide explains the interactive controls landed by **PRP-37 — Forecast -Intelligence C** (the operator-facing surface for the V2 feature contract and -the model zoo introduced by PRP-35 and PRP-36). It is RAG-indexable: ask the -Chat agent any question about model families, feature packs, horizon buckets, -or champion/challenger workflows and it will cite this document. +> **Moved.** This guide has been absorbed into the [user manual](../manual/README.md). -## Model families +Its content now lives in: -ForecastLabAI groups its models into three families. The Family is a -property of the model code, not a label you pick — it is what the segmented -**Family** Tabs control on `/visualize/forecast` and `/visualize/backtest` -filter the Model Select against. +- **[Forecasting](../manual/analyst/forecasting.md)** — model families, the V1/V2 feature frame, the eleven feature packs and their defaults, safety classes, and feature importance. +- **[Backtesting](../manual/analyst/backtesting.md)** — per-horizon-bucket metrics, the baseline-versus-feature-aware comparison, and the batch sweep presets. +- **[Champion selector](../manual/analyst/champion-selector.md)** — champion compatibility, stale-alias reasons, and the Promote dialog's three gates. -| Family | Members | When it shines | -|----------|----------------------------------------------------------------------------------------------------|----------------| -| Baseline | `naive`, `seasonal_naive`, `moving_average`, `weighted_moving_average`, `seasonal_average` | Sanity check, target-only history, very short windows | -| Tree | `regression` (HistGBR), `lightgbm`, `xgboost`, `random_forest` | Mid-to-long horizons with rich feature signal | -| Additive | `prophet_like` (Ridge additive), `trend_regression_baseline` | Strong yearly seasonality, interpretable coefficients | +Related chapters: -Baselines do **not consume features**. Tree and additive families do — and only -those families surface the V2 feature-frame option. - -## Feature frame: V1 vs V2 - -The **Feature frame** Select is the second control in the Train-a-new-model -row. It chooses how the model sees the past. - -- **V1 — target-only.** The classic lags + same-DOW mean. Every model in - every family can train on V1. -- **V2 — feature-aware.** The PRP-35 contract. Adds eleven optional - *feature packs* (see below). Available for tree and additive families only; - baselines reject it with a tooltip explanation. - -The backend default is V1; the UI only sends `feature_frame_version=2` when -the operator explicitly picks V2. A V1 train with `feature_groups` is -rejected by the backend with a 422. - -## Feature packs (V2 only) - -When V2 is picked, the **Feature packs** toggle row appears. Each pack is a -named subset of the V2 feature columns: - -| Pack ID | What it carries | -|----------------------|------------------| -| `target_history` | Lag features and same-day-of-week mean | -| `rolling` | Rolling means over multiple windows | -| `trend` | 30-day and 90-day trend | -| `calendar` | Day-of-week, month, sin/cos calendar signals | -| `price_promo` | Price level and promotion indicators | -| `inventory` | On-hand stock and stockout flags | -| `lifecycle` | Product lifecycle stage | -| `replenishment` | Inbound stock cadence | -| `returns` | Return intensity | -| `exogenous_weather` | Weather signals (when seeded) | -| `exogenous_macro` | Macro signals (when seeded) | - -Use the **Use defaults** button to load the six packs the V2 contract uses by -default (`target_history`, `calendar`, `rolling`, `trend`, `price_promo`, -`lifecycle`). The **Clear** button removes every pack; submitting with an -empty selection forwards `feature_groups: undefined` to the backend (treated -as the default set on the server). - -A pack may carry a per-row safety chip (`Safe`, `Conditionally safe`, -`Requires supplied data`). The chip is rendered when the server returns a -`feature_safety_classes` map for the run. A `Requires supplied data` chip -means the pack reads a column the production pipeline must supply (e.g. -inventory or replenishment) — promote a run that uses it only if your -production pipeline can keep that column populated. - -## Per-horizon-bucket metrics - -The backtest visualization now surfaces a **Per-horizon-bucket** card under -the existing fold-metric chart, rendered only when the response carries -`bucketed_aggregated_metrics`. It splits the forecast error by horizon -distance: - -| Bucket id | Horizon range | -|-------------|----------------| -| `h_1_7` | Days 1-7 | -| `h_8_14` | Days 8-14 | -| `h_15_28` | Days 15-28 | -| `h_29_plus` | Days 29+ | - -Empty buckets are dropped from the response. Unknown bucket ids (a forward- -compatible bucket from a newer backend) are appended to the end of the table -alphabetically. - -Pick the displayed metric (MAE / sMAPE / WAPE / Bias / RMSE) with the -Select to the right of the card title. **RMSE** is a key inside the -`aggregated_metrics` dict — surfaced as a fourth tile on the Aggregated -Metrics card when the backend emits it. - -## Baseline vs feature-aware comparison - -When the backtest response carries `baseline_results` (a non-empty list of -ModelBacktestResult rows), a **Baseline vs feature-aware** table renders -below the bucket card. Every baseline runs on the **same folds, identical -splits** as the main model — so MAE / sMAPE / WAPE / RMSE comparisons are -apples-to-apples. Lower wins. - -## Champion compatibility - -Two runs are **comparable** for champion/challenger evaluation iff -ALL three hold: - -1. Same grain (`store_id`, `product_id`). -2. Overlapping data windows. -3. Same `feature_frame_version` (legacy runs without the field default to V1). - -The Compare runs page renders a **Champion compatibility** badge that -surfaces the verdict, and the metrics diff table adds a **Feature frame -version** row when at least one of the two runs declares it. - -## Stale aliases - -The Control Center page now surfaces stale aliases as their own card with a -**Reason** chip per row: - -| Reason chip | What it means | -|-----------------------------------|-----------------------------------------------------------------------| -| `newer success run` | A newer successful run for this grain has landed. | -| `artifact not verified` | The alias's run artifact failed SHA-256 verification. | -| `run not success` | The alias is pointing at a non-success run (failed or archived). | -| `V mismatch` | The newest comparable run uses a different `feature_frame_version`. | - -Alongside each chip, the row shows the **Alias V** and **Comparable V** -columns so the operator can read the version drift at a glance. - -## Safer Promote dialog - -The Control Center's **Promote** action now opens a confirmation dialog that -gates the promotion on three conditions: - -1. **Artifact verifies.** The dialog auto-fetches the candidate run's - SHA-256 verification result. A failure renders a red callout and the - Promote button stays disabled — no operator override. -2. **Worse-WAPE acknowledgement.** When the candidate's latest WAPE is - HIGHER than the current champion's, a red callout appears with the - exact deltas and a checkbox the operator must explicitly tick. -3. **Feature-frame-version mismatch acknowledgement.** When the candidate's - `feature_frame_version` differs from the champion's, an amber callout - warns that the alias's feature contract will silently change. A - checkbox the operator must tick releases the Promote button. - -The alias name input remains; the dialog defaults the alias to -`production`. Cancel preserves no state — both acknowledgements reset. - -## Batch sweep presets - -The Batch Runner page now hosts a **Sweep preset** Select with five built-in -presets. Picking a preset overwrites the matrix; the matrix can still be -hand-edited afterward. - -| Preset | What it loads | -|---------------------------------|---------------| -| Quick baseline sweep | All five baseline models on V1 | -| Feature-aware comparison | Regression / LightGBM / XGBoost / RandomForest / Prophet-like on V2 with default packs | -| Champion/challenger refresh | Champion + strongest challenger from the registry (supplied by the page) | -| Stockout-sensitive products | Regression on V2 with the inventory + replenishment + returns packs | -| High-WAPE recovery | Every feature-aware model on V2 with default packs | - -Below the preset Select is the **Sweep matrix** picker — a checkbox grid of -model × V1/V2. Toggling a V2 cell adds a per-row feature-packs editor below -the grid. The matrix caps at 24 rows by default (configurable on the -picker). - -## Anti-patterns - -- **Do not** pick V2 for a baseline model — V2 has no effect on a model that - ignores features. The UI disables this combination with a tooltip. -- **Do not** promote a worse run without checking the explicit - acknowledgement checkbox. The gate exists for a reason. -- **Do not** promote across a feature-frame-version boundary without - verifying your production pipeline supplies the columns the new V demands. -- **Do not** read RMSE from `aggregated_metrics["rmse"]` for old jobs — - RMSE landed in PRP-36, and pre-PRP-36 backtest jobs in the registry will - not carry it. The UI omits the RMSE tile in that case. +- [Seeding data](../manual/operator/seeding-data.md) — why a feature pack whose signal was never seeded contributes nothing. +- [Extending ForecastLabAI](../manual/integrator/extending.md) — adding a model type or a feature pack. diff --git a/docs/user-guide/agents-and-rag-guide.md b/docs/user-guide/agents-and-rag-guide.md index 052d8b33..3d7f834f 100644 --- a/docs/user-guide/agents-and-rag-guide.md +++ b/docs/user-guide/agents-and-rag-guide.md @@ -1,121 +1,10 @@ # Agents and RAG Guide -ForecastLab includes a conversational AI layer — chat agents — backed by a -**RAG knowledge base** (retrieval-augmented generation). This guide explains how both -work and how to use them safely. +> **Moved.** This guide has been absorbed into the [user manual](../manual/README.md). -## The RAG Knowledge Base +Its content now lives in **[Chat and knowledge](../manual/analyst/chat-and-knowledge.md)** — the two agents, the RAG indexing and retrieval pipeline, the human-in-the-loop approval gate, session limits, and model fallback. -RAG lets the system answer questions using a body of indexed documents rather than -only the language model's general training. ForecastLab uses it to ground answers in -**project documentation**. +Related chapters: -### How indexing works - -When you index a document: - -1. The document is split into overlapping **chunks** (markdown is split by heading, - OpenAPI specs by endpoint). -2. Each chunk is converted into an **embedding** — a numeric vector capturing its meaning. -3. Chunks and embeddings are stored in PostgreSQL using the `pgvector` extension. - -Indexing is **idempotent**: each document is identified by its path and a content -hash, so re-indexing unchanged content does nothing, and changed content replaces the -old chunks cleanly. - -### How retrieval works - -A search query is embedded the same way, then compared against every stored chunk by -**cosine similarity**. The closest chunks above a similarity threshold are returned, -each with a relevance score and a citation back to its source document. Retrieval -returns evidence — passages — not a generated answer; the agent decides what to do -with them. - -### Using it - -- **Knowledge page** (`/knowledge`) — browse the indexed corpus and run live semantic - searches. -- **Admin → RAG Sources** — index a new document, list sources, or delete one. -- **API** — `POST /rag/index`, `POST /rag/retrieve`, `GET /rag/sources`, - `DELETE /rag/sources/{id}`. - -### Embedding providers - -Embeddings come from either **OpenAI** or a local **Ollama** server. The active -provider, model, and vector dimension are shown and changed under **Admin → AI models** -(`GET` / `PATCH /config/ai`). Local Ollama keeps document content off external services. - -## The Chat Agents - -The agents are conversational assistants built with PydanticAI. Two agent types exist: - -- **`rag_assistant`** — answers questions using the RAG knowledge base. -- **`experiment`** — can run forecasting experiments (training, backtesting, registry - actions) on your behalf. - -### Talking to an agent - -Use the **Chat** page (`/chat`) or the API: - -1. `POST /agents/sessions` — open a session, choosing the agent type. -2. `POST /agents/sessions/{id}/chat` — send a message and get the full response, or - connect to `WS /agents/stream` for token-by-token streaming. -3. `DELETE /agents/sessions/{id}` — close the session. - -A session keeps its message history, so the agent remembers earlier turns in the -conversation. - -### Tools - -Agents can call **tools** — typed functions that fetch data or perform actions -(retrieve documentation, list model runs, start a backtest, and so on). When an agent -uses a tool, the chat UI shows the call and its result, so you can see exactly how an -answer was produced. - -## The Human-in-the-Loop Approval Gate - -Most tools are read-only and run immediately. Tools that **change state** — for -example creating a registry alias or archiving a run — are different: they **pause and -wait for your approval**. - -When an agent wants to run one of these tools: - -1. The session enters an `awaiting_approval` state and an `approval_required` event is - emitted. -2. Nothing happens until you respond. -3. You approve or reject via `POST /agents/sessions/{id}/approve` (the Chat page - surfaces this as a prompt). -4. On approval the tool runs; on rejection it is skipped. - -This gate means an agent can never silently mutate the model registry — a person is -always in the loop for consequential actions. The set of approval-gated tools is a -deliberate, fixed list. - -### Other safety limits - -Each session is bounded so an agent cannot run away: - -- a **token budget** per session, -- a **maximum number of tool calls** per session, -- a **timeout** wrapping each agent run. - -The **Agent Guide** page (`/guide`) shows these limits live, along with the available -tools and example prompts. - -## Putting It Together - -A typical RAG-assisted exchange: you ask a question on the Chat page → the -`rag_assistant` agent calls its retrieval tool → the tool runs a semantic search over -the indexed corpus → the agent reads the returned passages → it answers, grounded in -real documentation, and you can see the citations. For experiments, the `experiment` -agent can additionally trigger training or backtesting — pausing for your approval -before anything that writes to the registry. - -## Tips - -- The agents need an LLM API key (`OPENAI_API_KEY` or `ANTHROPIC_API_KEY`) in `.env`. - Without one, the chat features are unavailable but the rest of the system still works. -- For useful RAG answers, index relevant documentation first — an empty corpus means - the assistant has nothing to cite. -- Watch the tool-call display in the chat: it is the simplest way to understand how - the agent reached its answer. +- [Configuration reference](../manual/configuration.md) — every RAG and agent setting, including the `agent_require_approval` gate and the embedding-dimension constraint. +- [API reference](../manual/integrator/api-reference.md) — the `/rag` and `/agents` endpoint groups and the WebSocket streams. diff --git a/docs/user-guide/champion-selector-guide.md b/docs/user-guide/champion-selector-guide.md index 1bef1afb..d35f44f5 100644 --- a/docs/user-guide/champion-selector-guide.md +++ b/docs/user-guide/champion-selector-guide.md @@ -1,126 +1,11 @@ # Champion Selector Guide -The **Champion Selector** turns "which forecasting model is best for this -store + product?" into a guided, end-to-end workflow: compare candidate models -on a leakage-safe backtest, read a recommendation, **decide** (accept it or -override), train the chosen model, generate and interpret its forecast, and — -only with explicit approval — **promote** it to a registry alias. +> **Moved.** This guide has been absorbed into the [user manual](../manual/README.md). -It lives at **`/visualize/champion`** in the dashboard and is served by the -`/model-selection/*` REST API (Swagger at **/docs** is the authoritative -contract). +Its content now lives in **[Champion selector](../manual/analyst/champion-selector.md)** — the select → compare → decide → train → forecast → interpret → promote workflow, the WAPE tie-break chain, the safety-stock heuristic, the promotion preconditions, and the three gates in the Promote dialog. -> **The golden rule of promotion:** the app *recommends* a champion, but a -> human *approves* it, and that decision is **recorded**. Promotion is never -> automatic. +Related chapters: -## The journey at a glance - -``` -Select → Run comparison → Results → Decide / override → Train → Forecast → Interpret → Promote -``` - -### 1 · Select & check availability - -Pick a store, a product, a time period, a forecast horizon (1–90 days), and the -candidate models to compare. The page checks **data availability** for the pair -and recommends a cross-validation split. A pair with too little history is -flagged *unusable* and the comparison is refused (`400`). - -### 2 · Run the comparison - -`POST /model-selection/runs` submits an asynchronous run (returns `202` with a -monitor URL); the page polls it to a terminal state. Each candidate is -backtested with time-series cross-validation; results are ranked deterministically. - -**Ranking** is by **WAPE** by default, with a fixed tie-break chain: -*WAPE, then sMAPE, then |bias|, then MAE.* The winner, runners-up, and any -failed candidates are all shown. - -### 3 · Decide — accept or override - -The recommended winner is pre-selected. You can: - -- **Accept the recommendation** → trains the ranked winner. -- **Override to another candidate** → you must confirm an explicit warning (the - recommended model and the WAPE gap are named) and may record a reason. The - override is flagged (`is_override=true`) and audited. A candidate that *failed* - its backtest is still override-trainable (training is independent of backtesting). - -`POST /model-selection/{id}/train-selected` trains the chosen model; -`train-winner` trains the recommendation. - -### 4 · Forecast - -`POST /model-selection/{id}/predict` generates the horizon forecast for the -trained model. The response carries the **peak** and **low** demand days plus a -**decision** block (see below). - -> **Capability limit.** A *feature-aware* model (`regression`, `prophet_like`, -> `lightgbm`, `xgboost`, `random_forest`) cannot auto-forecast here — it needs a -> future feature frame. The page shows a blocked state and routes you to the -> **What-If Planner** (Scenarios) instead of faking a forecast. - -### 5 · Interpret - -The **business interpretation** panel restates *why the model won*, the -**expected demand over the lead time**, and the **bias risk**: - -> Positive bias means the model under-forecasts (risk of stockouts); negative -> bias means it over-forecasts (risk of overstock). - -The **safety stock** panel shows a clearly-labeled, deterministic heuristic: - -``` -safety_stock = z(service_level) · σ_daily · √(lead_time_days) -expected_demand = average_demand · lead_time_days -reorder_point = expected_demand + safety_stock -``` - -`σ_daily` is the standard deviation of the daily forecast; `z` comes from a fixed -service-level table (90% → 1.2816, 95% → 1.6449, 97.5% → 1.9600, 99% → 2.3263), -snapping to the nearest level in between. Adjust the lead time / service level and -recompute. - -> **This is a heuristic** (demand variability only, constant lead time) — not a -> full inventory-optimisation model, and it **never** influences the model -> ranking. - -### 6 · Promote (approval-gated, audited) - -`POST /model-selection/{id}/promote` registers the trained model as a registry -`model_run` (transitioned to **SUCCESS** with a verified artifact) and points a -**registry alias** at it. It records a `promotion_decision` audit -(`approved_by`, the alias, the run id, the decision, the reason, and whether it -was an override). - -Promotion requires: - -- a valid **alias name** (`^[a-z0-9][a-z0-9\-_]*$`) — a bad name is rejected `422`; -- an **approver** (`approved_by`) — promotion is never anonymous; -- for a **non-recommended** (override) model, an explicit - `acknowledge_non_recommended=true` — else `422`; -- a **trained** model first — promoting before training is `422`. - -Re-promoting the same alias name repoints the existing alias (registry upsert -semantics). **Compare and promote stay separate** — promote performs no -ranking or comparison; it only registers and aliases the already-trained champion. - -## Endpoint reference - -| Method | Path | Purpose | -|--------|------|---------| -| POST | `/model-selection/runs` | Submit an async comparison (202) | -| GET | `/model-selection/{id}` | Poll progress / fetch terminal results | -| POST | `/model-selection/{id}/train-winner` | Train the ranked winner | -| POST | `/model-selection/{id}/train-selected` | Train a chosen candidate (override) | -| POST | `/model-selection/{id}/predict` | Forecast + inventory decision | -| POST | `/model-selection/{id}/promote` | Promote to a registry alias (audited) | - -## Notes & caveats - -- Backtest accuracy reflects historical fit, not a guarantee of future - performance; metrics measure correlation with past demand, not causation. -- The decision layer is **deterministic** — no LLM is involved. -- V2 (richer feature frame) runs promote as V2: the registry run records the - real `feature_frame_version`. +- [Backtesting](../manual/analyst/backtesting.md) — what the ranking metrics mean. +- [Artifacts and the registry](../manual/integrator/artifacts-and-registry.md) — artifact verification and alias semantics. +- [Demand and planning](../manual/analyst/demand-and-planning.md) — where a blocked feature-aware forecast routes you. diff --git a/docs/user-guide/dashboard-guide.md b/docs/user-guide/dashboard-guide.md index c12f27ff..d32c8471 100644 --- a/docs/user-guide/dashboard-guide.md +++ b/docs/user-guide/dashboard-guide.md @@ -1,120 +1,13 @@ # Dashboard Guide -The ForecastLab dashboard is a React web app at **http://localhost:5173**. This guide -walks through every page. The top navigation bar groups pages as: Dashboard, -Showcase, **Explorer** (menu), **Visualize** (menu), Knowledge, Chat, Agent Guide, -and Admin. A light/dark theme toggle sits on the right. +> **Moved.** This guide has been absorbed into the [user manual](../manual/README.md). -## Dashboard (`/`) +Its content now lives in **[Dashboard tour](../manual/analyst/dashboard-tour.md)**, which walks every page grouped as the navigation groups them, and adds a "which page answers which question" index. -The landing page. It shows headline **KPI cards** — total revenue, units sold, -transactions, average unit price, average basket — plus a revenue-over-time chart. -Use it for a quick health check of the seeded dataset. If the database is empty, -the cards read zero; seed data first (see Admin, or run `make demo`). +The deeper analytical pages have their own chapters: -## Showcase (`/showcase`) - -Runs the **end-to-end demo pipeline live in your browser**. Click to start, and the -page streams one status card per step: seed → features → train three models → -backtest → register the winner → alias → agent check. Each card flips to a -pass / fail / skip state, and a summary banner reports the winning model and its -accuracy. This is the best page for a guided demo of the whole system. - -Tip: tick **Re-seed first** if the database is empty or stale. Only one pipeline can -run at a time. - -## Explorer - -The Explorer menu contains read-only pages for browsing the underlying data and -model history. Tables support pagination, filtering, search, and sorting; clicking a -row opens a detail page. - -- **Sales** (`/explorer/sales`) — browse daily sales records. -- **Stores** (`/explorer/stores`) — list of retail stores. Click a store to open its - **detail page**: an entity profile, date-scoped KPIs, a revenue-over-time chart, - and a top-products drilldown. -- **Products** (`/explorer/products`) — list of products (SKUs). Click a product for - its **detail page**: profile, KPIs, revenue and lifecycle-demand curves, and a - top-stores drilldown. -- **Model Runs** (`/explorer/runs`) — every trained model tracked in the registry. - A **Family** badge column distinguishes baseline, tree, and additive models at - a glance. The run **detail page** shows configuration, metrics, runtime info, - cross-links to the store/product, an artifact-integrity check, a compare link, - and (for non-baseline runs) the canonical feature columns plus a feature - importance panel — see - [Advanced Model Metadata](./feature-reference.md#advanced-model-metadata) in the - Feature Reference for the data model and error semantics. The detail page also - hosts a **Feature frame** panel that renders V1/V2 + per-group columns + - per-column safety classes when the run carries that metadata (PRP-35/36). - Two runs can be compared side by side: a **Champion compatibility** badge - surfaces the comparable-run verdict (same grain + overlapping data windows + - same feature_frame_version), and the metrics-diff table now includes a - **Feature frame version** row. -- **Jobs** (`/explorer/jobs`) — submitted train/predict/backtest jobs. A job - **detail page** shows parameters, result JSON, error details, the linked run, a - cancel action, and live status polling. - -## Visualize - -The Visualize menu holds the analytical, chart-heavy pages. - -- **Demand Planner** (`/visualize/demand`) — rolls completed `predict` jobs into a - multi-SKU table showing tomorrow / next-week / next-month demand and the - inventory required to cover it. Includes a lead-time selector and a single-SKU - drill-in. Answers "how much will this SKU sell, and do I have enough stock?" -- **Forecast** (`/visualize/forecast`) — visualizes a model's horizon predictions. - The top of the page now also hosts a **Train a new model** card: a segmented - family picker (Baseline / Tree / Additive), a model-type Select filtered by the - picked family, a Feature frame V1/V2 Select, and (when V2 is picked) a feature- - pack toggle group. See [Advanced Forecasting Guide](./advanced-forecasting-guide.md). -- **Backtest Results** (`/visualize/backtest`) — charts backtest folds and the - accuracy metrics (MAE, sMAPE, WAPE, bias, stability) for a model run. When the - backtest response carries per-horizon-bucket metrics, a separate **Per-horizon- - bucket** card surfaces those (`Days 1-7 / 8-14 / 15-28 / 29+`) and a metric - switcher (MAE / sMAPE / WAPE / Bias / RMSE). When the response carries - baseline competitors, a **Baseline vs feature-aware** comparison table renders. -- **What-If Planner** (`/visualize/planner`) — the existing scenario simulation - view; impact card now carries a **method badge** - (`model-driven re-forecast` vs `heuristic adjustment`) so the planner - always sees how the scenario was produced. -- **Batch Runner** (`/visualize/batch`) — the existing batch runner now hosts a - **Sweep preset** Select (5 presets — quick baseline sweep, feature-aware - comparison, champion/challenger refresh, stockout-sensitive products, high-WAPE - recovery) and a **Sweep matrix** picker (multi-model × V1/V2). Picking a preset - prefills the matrix; rows can still be hand-edited. - -## Knowledge (`/knowledge`) - -Surfaces the **RAG knowledge base**: the indexed document corpus, a live semantic -search box, and current system state. Type a question to retrieve the most relevant -documentation passages with similarity scores. If the corpus is empty, the page -shows an empty state until documents are indexed (see the Admin page). - -## Chat (`/chat`) - -The **AI agent chat**. Ask questions in natural language; the assistant streams its -answer token by token and shows any tools it calls. Some actions pause for your -approval before they run. See the Agents and RAG Guide for details. - -## Agent Guide (`/guide`) - -An in-app reference for the chat agents: the tools they can use, the human-in-the-loop -approval gate, live session limits, and example prompts to try. - -## Admin (`/admin`) - -Operational controls, organized into tabs: - -- **Data seeding** — generate synthetic retail data from named scenarios, append more, - verify integrity, or clear the dataset. -- **RAG Sources** — list indexed knowledge documents, index a new document, and - delete sources. -- **Aliases** — manage model registry aliases (e.g. promote a run to `production`). -- **AI models** — view and change the agent LLM and RAG embedding configuration - live, with per-provider health indicators. - -## Notes - -- Pages fetch data from the backend API; if everything shows "Loading…", confirm the - backend is running and `VITE_API_BASE_URL` points at it. -- Explorer detail pages are reached by clicking table rows — they are not in the nav. +- [Forecasting](../manual/analyst/forecasting.md) — the Train-a-new-model card, model families, and the V1/V2 feature frame. +- [Backtesting](../manual/analyst/backtesting.md) — fold charts, horizon buckets, and the baseline comparison table. +- [Champion selector](../manual/analyst/champion-selector.md) — the compare → decide → train → promote workflow. +- [Demand and planning](../manual/analyst/demand-and-planning.md) — the Demand Planner and What-If Planner. +- [Chat and knowledge](../manual/analyst/chat-and-knowledge.md) — the Chat, Knowledge, and Agent Guide pages. diff --git a/docs/user-guide/feature-reference.md b/docs/user-guide/feature-reference.md index 521f795b..57d91e72 100644 --- a/docs/user-guide/feature-reference.md +++ b/docs/user-guide/feature-reference.md @@ -1,207 +1,15 @@ # Feature Reference -This is a capability-by-capability reference for ForecastLab's backend. Every feature -is a REST API served at **http://localhost:8123**; the interactive Swagger UI at -**/docs** is the authoritative, always-current contract. All errors use the RFC 7807 -`application/problem+json` format. +> **Moved.** This reference has been absorbed into the [user manual](../manual/README.md). -## Health +> **Note:** the model list in the previous version of this file was out of date — it named seven model types, and the system has **eleven**. The manual is generated against `app/shared/model_taxonomy.py`, the authoritative source. -- `GET /health` — liveness probe; returns `{"status": "ok"}`. +Its content now lives in: -## Data Platform and Ingest +- **[API reference](../manual/integrator/api-reference.md)** — the shared conventions, the RFC 7807 error envelope, and all twenty endpoint groups. +- **[Forecasting](../manual/analyst/forecasting.md)** — the eleven model types, three families, the V1/V2 feature frame, the eleven feature packs, and how to read feature importance. +- **[Artifacts and the registry](../manual/integrator/artifacts-and-registry.md)** — run lifecycle, artifact verification, and alias semantics. +- **[Champion selector](../manual/analyst/champion-selector.md)** — the `/model-selection/*` workflow. +- **[Data model](../manual/integrator/data-model.md)** — the twenty-three tables and the slices that own them. -The data platform owns seven retail tables: `store`, `product`, `calendar`, -`sales_daily`, `price_history`, `promotion`, and `inventory_snapshot_daily`. - -- `POST /ingest/sales-daily` — batch-load daily sales. Resolves natural keys - (store code, SKU) to IDs and upserts idempotently, so re-sending the same batch is - safe. - -## Dimensions - -Reference data — the "who" and "what" behind the sales facts. - -- `GET /dimensions/stores` — list stores (pagination, region / store-type filters, - case-insensitive search, optional sorting). -- `GET /dimensions/stores/{store_id}` — one store by ID. -- `GET /dimensions/products` — list products (category / brand filters, SKU / name - search, optional sorting). -- `GET /dimensions/products/{product_id}` — one product by ID. - -## Analytics - -Read-only aggregates computed over the sales data. - -- `GET /analytics/kpis` — headline KPIs: revenue, units, transactions, average unit - price, average basket. -- `GET /analytics/drilldowns` — group sales by store, product, category, region, or date. -- `GET /analytics/timeseries` — period-bucketed sales series (day / week / month / - quarter) for revenue-over-time charts. -- `GET /analytics/inventory-status` — latest inventory snapshot per store-product pair. - -## Feature Engineering - -Turns raw sales into model-ready features while strictly preventing **data leakage** — -features never use information from the future. - -- `POST /featuresets/compute` — compute time-safe features (lags, rolling-window - statistics, calendar effects) up to a cutoff date. -- `POST /featuresets/preview` — preview computed features with sample rows. - -## Forecasting - -Trains demand-forecasting models and generates predictions. - -- `POST /forecasting/train` — train a model. Supported model types: - - **Baselines**: `naive`, `seasonal_naive`, `moving_average` — always available. - - **Tree (feature-aware)**: `regression` (HistGradientBoostingRegressor, always - available), `lightgbm` (requires the `ml-lightgbm` extra), `xgboost` (requires - the `ml-xgboost` extra). - - **Additive (feature-aware)**: `prophet_like` — a Ridge regressor over the - canonical 14-column feature frame; always available. -- `POST /forecasting/predict` — generate horizon predictions from a trained model. -- `GET /forecasting/runs/{run_id}/feature-metadata` — return the canonical - feature columns the trained model consumed and the fitted estimator's native - feature importance (tree models) or signed coefficients (additive - `prophet_like`). See **Advanced Model Metadata** below. -- `GET /forecasting/jobs/{job_id}/feature-metadata` — the job-keyed sibling of - the run endpoint; use it from the forecast viz page, which only holds a - `job_id`. - -The three baselines exist as honest comparison points — a machine-learning model is -only worth using if it beats them. - -### Advanced Model Metadata - -Every model returned by `/registry/runs` carries a computed `model_family` field -— `baseline` (naive / seasonal_naive / moving_average), `tree` (regression / -lightgbm / xgboost), or `additive` (prophet_like). The dashboard surfaces this -in four places: a **Family** badge column on the runs explorer, a badge + cards -on the run detail page (the 14 canonical feature columns and a feature -importance panel), a side-by-side comparison on the run compare page (rendered -only when both runs share a non-baseline family), and a collapsible importance -panel on the forecast viz page tied to the train job. - -For a `tree`-family run the panel renders non-negative bars whose length -reflects relative magnitude (the boosters' native `feature_importances_` -attribute — LightGBM's `'split'`, XGBoost's `'weight'`, etc.; the label is -shown at the top of the panel). For an `additive` (`prophet_like`) run the -panel preserves the **sign** of the Ridge coefficient — a positive coefficient -renders green with a `TrendingUp` icon; a negative one renders red with -`TrendingDown`. - -> **Correlation, not causation.** Feature importance is model-derived. It -> reflects how much each feature reduced the model's training error — not -> real-world causation. Two products with similar importance profiles are not -> necessarily driven by the same business factors. - -Three error semantics map cleanly to RFC 7807 `application/problem+json`: -- **400 `BAD_REQUEST`** — the run is a baseline (no native importance vector - exists). The panel renders a neutral muted message. -- **404 `NOT_FOUND`** — the run or job is not in the registry. -- **422 `UNPROCESSABLE_ENTITY`** — the run has no artifact yet - (`pending` / `running` / `failed`), the artifact file has been deleted from - disk, an optional `ml-*` extra is not installed at unpickle time, or the - underlying estimator does not expose `feature_importances_` (sklearn's - `HistGradientBoostingRegressor`, used by `regression`, does not). - The 422 type URI is `UNPROCESSABLE_ENTITY` (distinct from `VALIDATION_ERROR`, - which is reserved for input failures). - -## Backtesting - -Measures how accurate a model would have been, using time-series cross-validation. - -- `POST /backtesting/run` — run rolling or expanding train/test splits and report - accuracy metrics: **MAE**, **sMAPE**, **WAPE**, **bias**, and **stability**. - -## Model Registry - -Tracks every trained model so runs are reproducible and comparable. - -- `POST /registry/runs` — create a model run record (starts `pending`). -- `GET /registry/runs` — list runs with filters, pagination, and sorting. -- `GET /registry/runs/{run_id}` — run details, including metrics and runtime info. -- `PATCH /registry/runs/{run_id}` — update a run's status, metrics, or artifact location. -- `GET /registry/runs/{run_id}/verify` — verify the model artifact's SHA-256 integrity. -- `GET /registry/compare/{run_id_a}/{run_id_b}` — diff two runs. -- `POST /registry/aliases` — create or move an alias (e.g. `production`); aliases may - point only to a successful run. -- `GET /registry/aliases`, `GET /registry/aliases/{name}`, `DELETE /registry/aliases/{name}` - — manage aliases. - -A run moves through `pending → running → success` (or `failed`), and an alias is a -human-friendly pointer (like `production` or `champion`) to a chosen successful run. - -## Champion Selector - -An end-to-end "which model is best, and now what?" workflow over one (store, -product) pair: compare candidate models, accept or override the recommendation, -train, forecast, interpret, and promote to a registry alias. - -- `POST /model-selection/runs` — submit an async candidate comparison (`202`). -- `GET /model-selection/{id}` — poll progress / fetch the ranked results + winner. -- `POST /model-selection/{id}/train-winner` — train the ranked winner. -- `POST /model-selection/{id}/train-selected` — train a chosen candidate (override + audit). -- `POST /model-selection/{id}/predict` — forecast the trained model + a labeled - safety-stock decision heuristic (feature-aware models are blocked → use Scenarios). -- `POST /model-selection/{id}/promote` — approval-gated, audited promotion to a - registry alias (requires an approver; a non-recommended model needs an explicit ack). - -See the full walkthrough in **[champion-selector-guide.md](./champion-selector-guide.md)**. - -## Jobs - -Long-running work — training, prediction, backtesting — submitted as jobs. - -- `POST /jobs` — submit a `train`, `predict`, or `backtest` job; returns a `job_id`. -- `GET /jobs` — list jobs with filters and sorting. -- `GET /jobs/{job_id}` — job status and result JSON. -- `DELETE /jobs/{job_id}` — cancel a pending job. - -## RAG Knowledge Base - -Semantic search over indexed documents. See the Agents and RAG Guide for the full -picture. - -- `POST /rag/index` — index a markdown or OpenAPI document; idempotent via content hash. -- `POST /rag/retrieve` — semantic search; returns the top-k most relevant passages. -- `GET /rag/sources` — list indexed sources. -- `DELETE /rag/sources/{source_id}` — delete a source and its chunks. - -## Agents - -The conversational AI layer. See the Agents and RAG Guide. - -- `POST /agents/sessions` — open a chat session (`experiment` or `rag_assistant`). -- `GET /agents/sessions/{id}` — session status and message history. -- `POST /agents/sessions/{id}/chat` — send a message; returns the full response. -- `POST /agents/sessions/{id}/approve` — approve or reject a pending tool call. -- `DELETE /agents/sessions/{id}` — close a session. -- `WS /agents/stream` — token-by-token streaming with tool-call events. - -## Seeder ("The Forge") - -Generates realistic synthetic retail data so you have something to forecast. - -- `GET /seeder/status` — current dataset state. -- `GET /seeder/scenarios` — available named scenarios. -- `GET /seeder/channels` — available sales channels. -- `POST /seeder/generate` — generate a dataset from a scenario. -- `POST /seeder/append` — append more data to an existing dataset. -- `DELETE /seeder/data` — clear the generated data. -- `GET /seeder/exogenous` — exogenous signal data. -- `POST /seeder/verify` — verify dataset integrity. - -## Demo Pipeline - -- `POST /demo/run` — run the full end-to-end pipeline in one call. -- `WS /demo/stream` — stream per-step events for the live Showcase page. - -## Configuration - -- `GET /config/ai` — effective AI-model configuration (agent LLM + RAG embeddings); - API keys are always masked. -- `PATCH /config/ai` — change AI-model settings live, with no restart. -- `GET /config/providers/health` — per-provider connectivity status. -- `GET /config/ollama/models` — models available on the configured Ollama host. +The authoritative, always-current API contract remains the generated OpenAPI schema at **http://localhost:8123/docs**. diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.md index 4f371c2a..4d116ffa 100644 --- a/docs/user-guide/getting-started.md +++ b/docs/user-guide/getting-started.md @@ -1,103 +1,10 @@ # Getting Started with ForecastLab -This guide takes you from a fresh clone to a running ForecastLab system with data, -trained models, and a working dashboard — in about ten minutes. +> **Moved.** This guide has been absorbed into the [user manual](../manual/README.md). -## What ForecastLab Is +Its content now lives in two chapters: -ForecastLab is a **retail demand-forecasting system** you run on a single machine. It -covers the whole forecasting lifecycle end to end: +- **[Installation](../manual/operator/installation.md)** — prerequisites, `.env`, Docker, `uv sync`, migrations, and the pass condition for each step. +- **[Quickstart](../manual/operator/quickstart.md)** — the fastest path to a working system with data and trained models. -1. **Data platform** — stores, products, calendar, daily sales, prices, promotions, inventory. -2. **Ingest** — load sales data through a batch API. -3. **Feature engineering** — build time-safe features (lags, rolling windows, calendar effects). -4. **Forecasting** — train baseline and machine-learning models. -5. **Backtesting** — measure accuracy with time-series cross-validation. -6. **Model registry** — track every trained model, compare runs, promote a champion. -7. **RAG knowledge base** — semantic search over project documentation. -8. **AI agents** — a chat assistant that can run experiments and answer questions. -9. **Dashboard** — a React web app that surfaces all of the above. - -It is built for learning, demos, and portfolio use. It is **not** a multi-tenant SaaS, -not a real-time streaming system, and needs no cloud account — everything runs locally. - -## Prerequisites - -- **Docker** (for the PostgreSQL database) -- **Python 3.12** with [`uv`](https://docs.astral.sh/uv/) (the Python package manager) -- **Node.js** with `pnpm` (enabled through `corepack`) - -## Install and Run - -Run these from the repository root. - -```bash -# 1. Configure environment — add your OpenAI / Anthropic API keys to .env -cp .env.example .env - -# 2. Start PostgreSQL + pgvector (listens on host port 5433) -docker compose up -d - -# 3. Install backend dependencies -uv sync --extra dev - -# 4. Apply database migrations -uv run alembic upgrade head - -# 5. Start the backend API (http://localhost:8123) -uv run uvicorn app.main:app --reload --port 8123 -``` - -In a second terminal, start the web dashboard: - -```bash -cd frontend -corepack enable pnpm -pnpm install -pnpm dev # dashboard at http://localhost:5173 -``` - -Open **http://localhost:5173** in your browser. The interactive API documentation -(Swagger UI) is available at **http://localhost:8123/docs**. - -## Load Data and See It Work - -A fresh database is empty. The fastest way to see the whole system in action is the -**end-to-end demo**, which seeds data, computes features, trains three models, -backtests them, registers the winner, and exercises the agent: - -```bash -make demo -``` - -You can also watch the same pipeline run live in the browser on the **Showcase** page -(see the Dashboard Guide). To generate data without the full pipeline, use the -**Admin** page or the seeder API directly. - -## Key Ports and URLs - -| Service | URL | -|----------------|------------------------------| -| Dashboard | http://localhost:5173 | -| Backend API | http://localhost:8123 | -| API docs | http://localhost:8123/docs | -| PostgreSQL | localhost:5433 | - -## If Something Goes Wrong - -- **Dashboard shows "Loading…" everywhere** — the frontend cannot reach the backend. - Check that the API is running (`curl http://localhost:8123/health`) and that - `frontend/.env` has `VITE_API_BASE_URL=http://localhost:8123`. -- **Database connection refused** — make sure `docker compose up -d` succeeded and - migrations are applied (`uv run alembic upgrade head`). -- **API keys** — the AI agent and RAG features need `OPENAI_API_KEY` and/or - `ANTHROPIC_API_KEY` set in `.env`. Forecasting and the dashboard work without them. -- **Browser dogfood / UI verification** — run `./scripts/dogfood-browser.sh` to - verify Playwright + snap chromium are ready for headless dashboard exercises; - pass a Python file path to execute it through the prepared environment. - -## Next Steps - -- **Dashboard Guide** — a tour of every page in the web app. -- **Feature Reference** — what each part of the system does and its API endpoints. -- **Agents and RAG Guide** — how the chat assistant and knowledge base work. +New readers should start at the [manual index](../manual/README.md), which splits the documentation into an operator, analyst, and integrator track. diff --git a/docs/user-guide/showcase-manual-demo-guide.md b/docs/user-guide/showcase-manual-demo-guide.md index c20efbf9..31297551 100644 --- a/docs/user-guide/showcase-manual-demo-guide.md +++ b/docs/user-guide/showcase-manual-demo-guide.md @@ -1,5 +1,10 @@ # Showcase Manual Demo Guide +> **A reviewer QA procedure, not user documentation** — kept here deliberately rather than +> folded into the [user manual](../manual/README.md). For running the system, see +> [Quickstart](../manual/operator/quickstart.md); for the dashboard, see the +> [Dashboard tour](../manual/analyst/dashboard-tour.md). + This guide describes how to manually review the ForecastLabAI `/showcase` experience from a clean or controlled local environment. It is intended for technical reviewers, maintainers, and users evaluating the product. It focuses diff --git a/docs/user-guide/showcase-walkthrough.md b/docs/user-guide/showcase-walkthrough.md index 0f63837e..3591ac19 100644 --- a/docs/user-guide/showcase-walkthrough.md +++ b/docs/user-guide/showcase-walkthrough.md @@ -1,5 +1,10 @@ # Showcase walkthrough +> **Roadmap document, not user documentation** — kept here deliberately rather than folded +> into the [user manual](../manual/README.md), because it describes unshipped behavior. +> The manual documents only what ships today: see +> [Quickstart](../manual/operator/quickstart.md#the-same-thing-in-the-browser). + > **Status:** Walkthrough draft. Sections marked **Planned (PRP-{N})** describe behavior the four-PRP `/showcase` upgrade epic will deliver — they are NOT in `dev` yet. Sections under "Quick start (current behavior)" and "What `/showcase` exercises today" describe the page as it ships on `dev` today. ## Overview From 310322a6accc681de85238058818e92865f4e0ed Mon Sep 17 00:00:00 2001 From: Gabor Szabo Date: Wed, 12 Aug 2026 07:15:34 +0200 Subject: [PATCH 2/2] docs(docs): correct manual claims found by live verification (#428) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the stack end to end and checked the quickstart chapter against real output. Four corrections: - /health returns {"status":"ok","database":null}, not {"status":"ok"}. Fixed in installation, troubleshooting, and api-reference. (The root README carries the same inaccuracy.) - `docker compose up -d` starts postgres, backend AND frontend — only ollama is profile-gated. Host-mode instructions now name the service: `docker compose up -d postgres`. - The demo trains three baselines (naive, seasonal_naive, moving_average), so its winner says nothing about feature-aware models. Removed the misleading implication and pointed at the champion selector for real comparisons. - reset/agent/cleanup legitimately report skip and the run still exits 0 GREEN. Documented skips as non-failures. Adds a troubleshooting entry for "Multiple head revisions are present", which breaks `alembic upgrade head`, the backend container's startup command, and `make demo`. Verified as correct: exit code 2 on precondition failure, the 11-step pipeline order, the summary-line format, SHA-256 artifact verification, graceful agent skip without an API key, and seed reproducibility — a re-run on seed 42 reproduced the row count, date range, selected pair, and WAPE to four decimals. --- docs/manual/integrator/api-reference.md | 2 +- docs/manual/operator/installation.md | 10 +++++++--- docs/manual/operator/quickstart.md | 19 +++++++++++++++---- docs/manual/operator/running-the-stack.md | 4 +++- docs/manual/troubleshooting.md | 15 ++++++++++++++- 5 files changed, 40 insertions(+), 10 deletions(-) diff --git a/docs/manual/integrator/api-reference.md b/docs/manual/integrator/api-reference.md index 5ba3b14c..b40c942b 100644 --- a/docs/manual/integrator/api-reference.md +++ b/docs/manual/integrator/api-reference.md @@ -81,7 +81,7 @@ Twenty routers. `/health` sits at the root; the rest are prefixed. | Prefix | Purpose | |---|---| -| `/health` | Liveness probe → `{"status":"ok"}`. | +| `/health` | Liveness probe → `{"status":"ok","database":null}`. Assert on `status`; `database` is a detail slot, `null` on a plain check. | | `/ingest` | Batch sales load — idempotent. | | `/dimensions` | Stores and products: list with filters, search, sorting, pagination; fetch by id. | | `/analytics` | Read-only aggregates: `kpis`, `drilldowns`, `timeseries`, `inventory-status`. | diff --git a/docs/manual/operator/installation.md b/docs/manual/operator/installation.md index 3fe0ace2..e08a0152 100644 --- a/docs/manual/operator/installation.md +++ b/docs/manual/operator/installation.md @@ -29,14 +29,16 @@ cp .env.example .env Every variable is documented in the [configuration reference](../configuration.md). Note that `.env.example` ships the commonly-changed subset, not the full surface. -## 2 · Start PostgreSQL + pgvector +## 2 · Start the database ```bash -docker compose up -d +docker compose up -d postgres ``` **Pass condition:** `docker compose ps` shows the Postgres service healthy, publishing host port **5433**. +Naming the service matters here. Plain `docker compose up -d` starts **all three** services — `postgres`, `backend`, and `frontend` — because only `ollama` is behind a profile. For a host-mode install you want just the database, since you are about to run the backend yourself in step 6. Starting the containerised backend as well is harmless but redundant, and it will fail its healthcheck if migrations are not yet applied. + Port 5433 is deliberate — it avoids colliding with a Postgres you may already run on 5432. The container still listens on 5432 internally; 5433 is only the host-side publication. ## 3 · Install backend dependencies @@ -80,9 +82,11 @@ uv run uvicorn app.main:app --reload --port 8123 ```bash curl http://localhost:8123/health -# {"status":"ok"} +# {"status":"ok","database":null} ``` +The `database` field is a detail slot that is `null` on a plain liveness check — `status: "ok"` is the part to assert on. Every response also carries an `x-request-id` header. + The interactive OpenAPI contract is now at **http://localhost:8123/docs**. That schema is generated from the code and is the authoritative API reference; this manual explains it rather than restating it. ## 7 · Start the dashboard diff --git a/docs/manual/operator/quickstart.md b/docs/manual/operator/quickstart.md index 90ed8f70..378d6edd 100644 --- a/docs/manual/operator/quickstart.md +++ b/docs/manual/operator/quickstart.md @@ -43,16 +43,27 @@ Each stage proves something specific: | Stage | What a pass proves | |---|---| | **precheck** | The API and database are reachable — the environment is real. | -| **seed** | The Forge can generate a reproducible dataset from seed 42. | +| **seed** | The Forge can generate a reproducible dataset from seed 42 (the `demo_minimal` scenario). | | **features** | Time-safe feature computation runs over the seeded history. | -| **train ×3** | Three model types fit and produce artifacts, concurrently. | +| **train ×3** | Three models fit and produce artifacts, concurrently. | | **backtest ×3** | Time-series cross-validation scores all three on identical folds. | | **register-winner** | The registry records the ranked winner with its metrics. | | **verify** | The winner's artifact passes SHA-256 verification. | -| **agent** | The agent layer answers through the live stack. | +| **agent** | The agent layer answers through the live stack — **skipped without an LLM API key**. | The comparison is meaningful precisely because the three models are backtested on the **same folds** — that is what makes "winner" a claim rather than an impression. +The three models the demo trains are all **baselines**: `naive`, `seasonal_naive`, and `moving_average`. The demo is a lifecycle smoke test, not a model bake-off — it proves the pipeline runs end to end, not that any particular model is good. To compare feature-aware models, use the [Champion selector](../analyst/champion-selector.md) or a [batch sweep](../analyst/backtesting.md#batch-sweeps). + +### Skips are not failures + +Several steps legitimately report `⏭️` rather than `✅`: + +- **reset** — skipped unless `--reset` is passed. +- **agent** and **cleanup** — skipped when no API key matches the configured `agent_default_model` provider. + +A run with skips still reports **GREEN** and exits `0`. That is the intended behavior: the agent layer is optional, and the demo says so rather than failing. + ## Reading the outcome A green run ends with a summary line naming the run count, the winning model, the alias, and wall-clock time — for example: @@ -61,7 +72,7 @@ A green run ends with a summary line naming the run count, the winning model, th runs=3 winner=seasonal_naive alias=demo-production wall_clock=87s ``` -**Which model wins is not fixed**, and a baseline winning is a legitimate result, not a bug. On synthetic data with a short history, `seasonal_naive` frequently beats feature-aware models — that is exactly why the baselines are in the comparison. +**Which model wins is not fixed.** It depends on the seeded data, and all three candidates here are baselines — so the winner is simply whichever baseline fits the generated series best. Do not read it as a claim about model quality in general. Exit codes: diff --git a/docs/manual/operator/running-the-stack.md b/docs/manual/operator/running-the-stack.md index a20d07f7..5e5557bd 100644 --- a/docs/manual/operator/running-the-stack.md +++ b/docs/manual/operator/running-the-stack.md @@ -33,12 +33,14 @@ Nearly every "works from my terminal but not in the container" problem is this t ## Host mode ```bash -docker compose up -d # Postgres only +docker compose up -d postgres # database only — name the service uv run alembic upgrade head uv run uvicorn app.main:app --reload --port 8123 # terminal 1 cd frontend && pnpm dev # terminal 2 ``` +**Name the service.** Plain `docker compose up -d` starts `postgres`, `backend`, **and** `frontend` — only `ollama` is profile-gated. In host mode you are running the backend and frontend yourself, so starting their containers too is redundant, and the backend container will fail its healthcheck until migrations are applied. + ## Container mode ```bash diff --git a/docs/manual/troubleshooting.md b/docs/manual/troubleshooting.md index cf3eaa83..ba68222f 100644 --- a/docs/manual/troubleshooting.md +++ b/docs/manual/troubleshooting.md @@ -17,7 +17,7 @@ Postgres (:5433) → migrations → backend API (:8123) → dashboard (:51 docker compose ps # is Postgres up and healthy? uv run python scripts/check_db.py # can the app reach and authenticate to it? uv run alembic current # are migrations applied? -curl http://localhost:8123/health # is the API up? → {"status":"ok"} +curl http://localhost:8123/health # is the API up? → {"status":"ok","database":null} ``` If `/health` answers `ok`, the backend and its database connection are both fine, and the problem is above that line — in the dashboard, or in the specific request you are making. @@ -39,6 +39,19 @@ Migrations were never applied, or a new migration landed after your last pull. R **The backend starts but every write fails after a `git pull`.** Same cause. Migrations are forward-only; pull then migrate. +**`Multiple head revisions are present for given argument 'head'`.** +Two migrations claim to be the tip, so `alembic upgrade head` cannot pick one and exits non-zero. This also fails the backend container's startup command and `make demo`, both of which run `upgrade head`. + +Almost always an uncommitted or newly-added migration branching off the same parent as an existing one. Diagnose and choose: + +```bash +uv run alembic heads # list the competing heads +uv run alembic upgrade # apply one specific head +uv run alembic upgrade heads # apply ALL heads (note the plural) +``` + +To resolve it properly rather than work around it, give one migration a `down_revision` pointing at the other so the history is linear again, or merge them with `alembic merge`. Do not edit a migration that is already merged to `dev` — add a new one. + ## Empty dashboard, no error **KPI cards all read zero and tables are empty.**