Expand PineForge Data documentation

CONTRIBUTING.md

@@ -4,6 +4,11 @@ Community data providers are the reason this repository exists. Keep the core
 contracts small and put vendor behavior behind Python provider modules. This
 repository does not accept a separate C++ provider implementation surface.
 
+User-facing architecture and API guides are indexed in
+[docs/index.md](docs/index.md). Update the applicable guide when changing public
+models, provider behavior, CLI options, report fields, runtime channels, or
+server/cache semantics.
+
 Docker is required for raw-Pine integration work. Engine and codegen are
 consumed only through the pinned `pineforge-release` image; do not add source
 submodules or duplicate their build logic here.

README.md

@@ -14,6 +14,22 @@ The engine does not import or link this package. Provider transport,
 authentication, retries, caching, symbol mapping, and vendor schemas stay here;
 the engine receives only normalized bars and ordered trades.
 
+## Documentation
+
+- [Documentation home](docs/index.md) — architecture, guarantees, and guide map.
+- [Getting started](docs/getting-started.md) — installation, first provider
+  request, and local or remote backtest.
+- [Normalized data model](docs/data-model.md) — instruments, contracts, bars,
+  live trades, macro vintages, and validation rules.
+- [Using providers](docs/providers.md) — registry, CCXT market discovery,
+  historical bars, live trades, configuration, and errors.
+- [Backtesting](docs/backtesting.md) — CLI options, configuration files, runtime
+  channels, report schema, and reproducibility.
+- [FastAPI server](docs/server.md) — concurrency, authentication, timeouts,
+  compile cache, and deployment.
+- [Provider contract](docs/provider-contract.md) — implementing and testing a
+  community exchange or broker adapter.
+
 ## Why Python first
 
 Provider integrations are dominated by HTTP, WebSockets, JSON, credentials,
@@ -189,31 +205,69 @@ hit/key/hash are included in response provenance. See
 [docs/server.md](docs/server.md) for endpoints, limits, deployment, and cache
 settings.
 
-## Development
+## Contributing
 
-```bash
-python3 -m venv .venv
-.venv/bin/pip install -e '.[dev,ccxt,server]'
-.venv/bin/ruff check .
-.venv/bin/mypy src
-.venv/bin/pytest
-PINEFORGE_DOCKER_TEST=1 .venv/bin/pytest tests/test_docker_integration.py
-```
+Community providers, market-model improvements, server/runtime work, tests, and
+documentation are welcome. Provider integrations are Python-only; engine and
+codegen changes belong in their upstream repositories and are consumed here
+through `pineforge-release`.
 
-## Provider boundary
+### Choose the right contribution path
 
-A provider adapter should:
+| Contribution | Primary location | Start here |
+|---|---|---|
+| Exchange or broker adapter | `src/pineforge_data/providers/` | [Provider contract](docs/provider-contract.md) |
+| Market, contract, bar, or request model | `src/pineforge_data/models.py`, `src/pineforge_data/requests.py`, `src/pineforge_data/providers/base.py` | Existing public models and protocols |
+| Backtest harness or HTTP client | `src/pineforge_data/cli/backtest.py`, `src/pineforge_data/server_client.py` | Harness unit tests |
+| FastAPI concurrency or compile cache | `src/pineforge_data/server.py`, `src/pineforge_data/compile_cache.py` | [Server guide](docs/server.md) |
+| Release-container integration | `src/pineforge_data/release_contract.py`, `src/pineforge_data/docker_runtime.py` | Pinned release contract and Docker tests |
+| Documentation or examples | `README.md`, `docs/` | A focused documentation PR |
 
-1. expose exact market discovery and symbol resolution;
-2. fetch or subscribe to its external service;
-3. retain source and instrument provenance;
-4. normalize timestamps to Unix milliseconds and records to the public models;
-5. emit stable ordering sequences when the provider supplies them;
-6. batch records before crossing the engine ABI when practical.
+For a new provider, implement the smallest applicable structural protocols,
+register its factory, keep its SDK in an optional dependency extra, and add
+offline fixture tests. Resolve exact upstream markets through their catalog;
+do not parse symbols to infer base, quote, settlement, or contract terms.
+Provider-specific fields stay in this repository and must not leak into
+`pineforge-engine`.
 
-It should not add provider-specific fields to `pineforge-engine`. Data that the
-engine does not consume remains in provider-owned metadata or higher-level
-models in this repository.
+### Development setup
+
+```bash
+git clone https://github.com/pineforge-4pass/pineforge-data.git
+cd pineforge-data
+python3 -m venv .venv
+.venv/bin/pip install -e '.[dev,ccxt,server]'
+```
 
-See [the provider contract](docs/provider-contract.md) and
-[CONTRIBUTING.md](CONTRIBUTING.md) before adding a provider.
+No Git submodules are required. Docker is needed only for release-runtime and
+end-to-end backtest work.
+
+### Before opening a pull request
+
+1. Keep the change focused and document any public API, report-schema, provider,
+   runtime-image, or cache-key compatibility impact.
+2. Add deterministic offline tests; CI must not require credentials or live
+   provider access.
+3. Keep credentials out of fixtures, logs, exception messages, and committed
+   configuration.
+4. Run the standard checks:
+
+   ```bash
+   .venv/bin/ruff format --check src tests
+   .venv/bin/ruff check .
+   .venv/bin/mypy src
+   .venv/bin/pytest
+   .venv/bin/python -m build
+   ```
+
+5. For Docker, FastAPI server, cache, or release-contract changes, also run:
+
+   ```bash
+   PINEFORGE_DOCKER_TEST=1 .venv/bin/pytest tests/test_docker_integration.py
+   ```
+
+Read the [documentation home](docs/index.md) and
+[CONTRIBUTING.md](CONTRIBUTING.md) for provider requirements,
+determinism rules, external provider entry points, and the complete checklist.
+For broad changes to public models or the report contract, open an issue first
+so providers and runtime consumers can agree on the shape before implementation.

docs/backtesting.md

@@ -0,0 +1,172 @@
+# Backtesting with PineForge Data
+
+The harness accepts raw PineScript. Data acquisition stays in Python while
+transpilation, compilation, and deterministic execution stay in
+`pineforge-release`.
+
+## Execution modes
+
+### Local release container
+
+Without a server URL, the harness pulls and runs the pinned release image
+locally. The container has networking disabled, a read-only root filesystem,
+dropped capabilities, `no-new-privileges`, a read-only input mount, and an
+executable `/tmp` tmpfs required for strategy compilation.
+
+### FastAPI server
+
+Set `--server-url` or `PINEFORGE_SERVER_URL` to send raw Pine and normalized
+bars to the concurrent server. Provider credentials and provider API calls stay
+on the harness host. Set `PINEFORGE_SERVER_API_KEY` for bearer authentication.
+
+The server returns synchronously while accepting multiple requests. Capacity,
+queueing, timeouts, authentication, deployment, and compiled-strategy caching
+are documented in the [server guide](server.md).
+
+## CLI reference
+
+```bash
+pineforge-backtest \
+  --pine strategy.pine \
+  --provider ccxt \
+  --venue kraken \
+  --symbol BTC/USD \
+  --timeframe 15m \
+  --start 2025-07-01T00:00:00Z \
+  --end 2025-07-08T00:00:00Z
+```
+
+| Group | Options |
+|---|---|
+| Strategy | `--pine`, `--strategy-params`, `--strategy-overrides` |
+| Data source | `--provider`, `--venue`/`--exchange`, `--symbol`, `--timeframe`, `--start`, `--end`, `--limit`, `--provider-config` |
+| Pine context | `--timezone`, `--session`, `--engine-timeframe`, `--script-timeframe` |
+| Fill modeling | `--bar-magnifier`, `--magnifier-samples` |
+| Local runtime | `--runtime-image`/`--image`, `--pull-policy`, `--execution-timeout` |
+| Remote runtime | `--server-url`, `--server-api-key-env`, `--execution-timeout` |
+| Output | `--output`, `--pretty` |
+
+`--start` and `--end` accept Unix milliseconds or timezone-aware ISO-8601. The
+end is exclusive. `--engine-timeframe` defaults to a Pine-compatible conversion
+of the provider timeframe, and `--script-timeframe` defaults to the engine
+timeframe.
+
+## Configuration files
+
+Provider constructor configuration:
+
+```json
+{
+  "enableRateLimit": true,
+  "timeout": 30000
+}
+```
+
+```bash
+--provider-config provider.json
+```
+
+Pine input overrides:
+
+```json
+{
+  "Fast Length": 8,
+  "Slow Length": 21
+}
+```
+
+```bash
+--strategy-params inputs.json
+```
+
+`strategy()` header overrides use a separate file:
+
+```json
+{
+  "default_qty_value": 5,
+  "commission_value": 0.04
+}
+```
+
+```bash
+--strategy-overrides overrides.json
+```
+
+Input and override values sent to the FastAPI service must be scalar strings,
+numbers, or booleans.
+
+## Runtime image policy
+
+The package default is an exact `pineforge-release` version and OCI digest. The
+`missing` pull policy downloads it only when absent; `never` supports offline
+runs; `always` refreshes a tag before running.
+
+The rolling channel is explicit:
+
+```bash
+--runtime-image ghcr.io/pineforge-4pass/pineforge-release:latest \
+--pull-policy always
+```
+
+Use the pinned default for reproducible research. A rolling tag may change
+engine or codegen behavior without a `pineforge-data` version change.
+
+## Report envelope
+
+The harness combines provider provenance with the release report:
+
+```json
+{
+  "schema_version": 1,
+  "request_id": null,
+  "provider": {
+    "name": "ccxt:kraken",
+    "adapter": "ccxt",
+    "venue": "kraken",
+    "source_timeframe": "15m",
+    "market": {
+      "symbol": "BTC/USD",
+      "provider_id": "XXBTZUSD",
+      "market_type": "spot",
+      "contract": null
+    }
+  },
+  "data": {
+    "requested_start_ms": 1751328000000,
+    "requested_end_ms": 1751932800000,
+    "first_bar_ms": 1751328000000,
+    "last_bar_ms": 1751931900000,
+    "bars": 672
+  },
+  "runtime": {
+    "mode": "local-container",
+    "release_image": "ghcr.io/...@sha256:..."
+  },
+  "backtest": {
+    "summary": {},
+    "trades": [],
+    "metrics": {},
+    "equity_curve": [],
+    "fingerprint": {}
+  }
+}
+```
+
+Remote responses include a request ID. Server runtime provenance also includes
+the generated C++ digest, compile-cache key/hit, and engine/codegen/release
+versions. Local runs record the resolved image digest and OCI component labels
+when Docker exposes them.
+
+## Reproducibility checklist
+
+Retain:
+
+- raw Pine source and strategy input/override files;
+- provider adapter and venue;
+- exact resolved symbol and provider market ID;
+- requested interval and normalized OHLCV snapshot or checksum;
+- release image reference and resolved digest;
+- report fingerprint and request ID.
+
+The pinned release currently does not expose trace collection. `--trace` fails
+explicitly rather than producing an incomplete report.