diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 76314d0..18bbb41 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,6 +52,11 @@ The callable receives `(venue, config)` and returns a structural `MarketDataProvider`. See [docs/provider-contract.md](docs/provider-contract.md) for the normalized model and a complete skeleton. +Every provider must have a second-level API guide at +`docs/providers/.md` and a catalog row in +`docs/providers.md`. Keep provider-specific installation, constructor, +configuration, behavior, errors, and limitations out of the catalog page. + ## Determinism and macro vintages Historical results must be reproducible. Cache or snapshot mutable upstream diff --git a/README.md b/README.md index e7a008d..8ad82e6 100644 --- a/README.md +++ b/README.md @@ -21,10 +21,8 @@ the engine receives only normalized bars and ordered trades. request, and local or remote backtest. - [Normalized data model](docs/data-model.md) — instruments, contracts, bars, live trades, macro vintages, and validation rules. -- [Using providers](docs/providers.md) — registry, CCXT market discovery, - historical bars, live trades, configuration, and errors. -- [Local files and databases](docs/local-data.md) — runtime schema discovery, - arbitrary CSV/SQLite DDL, SQLAlchemy databases, and column mappings. +- [Provider catalog](docs/providers.md) — shared lifecycle and second-level API + guides for CCXT, CSV, SQLite, and SQLAlchemy. - [Backtesting](docs/backtesting.md) — CLI options, configuration files, runtime channels, report schema, and reproducibility. - [FastAPI server](docs/server.md) — concurrency, authentication, timeouts, @@ -131,8 +129,9 @@ schema = await provider.inspect_schema() SQL identifiers are validated against reflected metadata; filter values are bound parameters. For complex transformations, expose a database view rather than putting raw SQL in harness configuration. See the complete -[local-data guide](docs/local-data.md) for CSV, native read-only SQLite, -SQLAlchemy URLs, timestamp handling, symbol/timeframe semantics, and CLI JSON. +[provider catalog](docs/providers.md), with dedicated API guides for +[CSV](docs/providers/csv.md), [SQLite](docs/providers/sqlite.md), and +[SQLAlchemy](docs/providers/sqlalchemy.md). ## Direct backtest harness diff --git a/docs/backtesting.md b/docs/backtesting.md index 863e2ea..fbbd3e3 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -99,7 +99,8 @@ The same harness accepts `--provider csv`, `--provider sqlite`, or `--provider sqlalchemy`. Their provider configuration maps arbitrary source columns to normalized OHLCV, while raw Pine and normalized bars follow the same local-container or FastAPI path. See -[Local files and databases](local-data.md) for complete configuration examples. +the [provider catalog](providers.md) for their individual API and configuration +guides. ## Runtime image policy diff --git a/docs/getting-started.md b/docs/getting-started.md index 22331ed..4e37509 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -103,7 +103,6 @@ cache. ## Next steps - [Learn the normalized data model](data-model.md). -- [Search and filter provider markets](providers.md). -- [Connect local CSV, SQLite, or SQLAlchemy data](local-data.md). +- [Choose a provider and open its API guide](providers.md). - [Configure parameters, runtime channels, and reports](backtesting.md). - [Deploy the concurrent server](server.md). diff --git a/docs/index.md b/docs/index.md index 68795a9..5901be3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -23,8 +23,7 @@ exchange / broker / macro API |---|---| | Install the package and run a first backtest | [Getting started](getting-started.md) | | Understand instruments, contracts, bars, trades, and macro vintages | [Data model](data-model.md) | -| Discover markets and use CCXT or another provider | [Using providers](providers.md) | -| Connect a CSV file, SQLite table, or SQLAlchemy database | [Local files and databases](local-data.md) | +| Choose a provider and open its API guide | [Provider catalog](providers.md) | | Configure local and remote raw-Pine backtests | [Backtesting](backtesting.md) | | Deploy and operate the concurrent FastAPI service | [FastAPI server](server.md) | | Implement a new exchange or broker adapter | [Provider contract](provider-contract.md) | diff --git a/docs/local-data.md b/docs/local-data.md deleted file mode 100644 index 3537ec6..0000000 --- a/docs/local-data.md +++ /dev/null @@ -1,296 +0,0 @@ -# Connect local files and databases - -PineForge Data can read historical OHLCV from a local CSV file, a SQLite table -or view, or any synchronous database supported by SQLAlchemy 2.x. The source -does not need to use PineForge column names or a prescribed DDL. - -The safe boundary is a runtime column mapping: - -```text -your file / table / view - ↓ reflect header or database metadata -timestamp, open, high, low, close, volume, [symbol], [timeframe] - ↓ validate and normalize -PineForge Bar records -``` - -PineForge Data never modifies a connected source. The native SQLite provider -opens the database in read-only mode. The SQLAlchemy provider performs -reflection and `SELECT` statements only; use a database account with read-only -permissions as defense in depth. - -## Runtime schema discovery - -Every tabular provider exposes `inspect_schema()` before data is loaded: - -```python -from pineforge_data import CsvBarProvider - - -async def inspect_file(): - provider = CsvBarProvider("./vendor-export.csv", venue="research") - schema = await provider.inspect_schema() - for column in schema.columns: - print(column.name, column.data_type, column.nullable) -``` - -`TabularSchema.infer_bar_mapping()` recognizes common names such as -`timestamp`, `datetime`, `open_time`, `PX_OPEN`, `qty`, `ticker`, and -`interval`. Supply only the unusual or ambiguous fields: - -```python -mapping = schema.infer_bar_mapping( - { - "timestamp": "Bucket Start", - "volume": "Total Traded Qty", - } -) -``` - -Inference never chooses between multiple plausible columns. A missing or -ambiguous field raises `SchemaMappingError` with the available columns and the -overrides needed to continue. For a completely custom schema, define all six -required fields explicitly: - -```python -from pineforge_data import BarColumnMapping - -mapping = BarColumnMapping( - timestamp="epoch seconds", - open="first px", - high="top px", - low="bottom px", - close="last px", - volume="traded qty", - symbol="security code", # optional - timeframe="bar interval", # optional -) -``` - -Provider constructors also accept a plain mapping. Plain mappings are partial -overrides; any fields not supplied are inferred at runtime. - -## Timestamp values - -Normalized timestamps are always Unix milliseconds. Set `timestamp_unit` for -numeric source values: - -| Value | Meaning | -|---|---| -| `seconds` or `s` | Unix seconds, including exact fractional milliseconds | -| `milliseconds` or `ms` | Unix milliseconds; the default | -| `microseconds` or `us` | Unix microseconds | -| `nanoseconds` or `ns` | Unix nanoseconds | -| `iso8601` | ISO-8601 text or database `datetime` objects | - -ISO-8601 values with an offset or `Z` are unambiguous. Naive text or database -datetimes use `timestamp_timezone`, which defaults to `UTC` and accepts an IANA -zone such as `America/New_York`. Values that cannot be represented exactly at -millisecond precision are rejected rather than rounded silently. - -## CSV - -CSV requires no optional dependency: - -```python -from pineforge_data import BarRequest, CsvBarProvider - - -async def load_csv(): - provider = CsvBarProvider( - "./exports/equity-bars.csv", - venue="research", - mapping={ - "timestamp": "Bucket Start", - "volume": "Total Traded Qty", - }, - timestamp_unit="iso8601", - ) - listing = await provider.resolve_market("AAPL") - return await provider.fetch_bars( - BarRequest( - listing.instrument, - timeframe="1m", - start_ms=1_751_328_000_000, - end_ms=1_751_414_400_000, - ) - ) -``` - -The default format is UTF-8 with a comma delimiter. Use `encoding` and -`delimiter` for other exports. CSV is scanned locally for each request; use a -database backend for large, repeatedly queried datasets. - -Harness configuration: - -```json -{ - "path": "/data/equity-bars.csv", - "timestamp_unit": "iso8601", - "columns": { - "timestamp": "Bucket Start", - "volume": "Total Traded Qty", - "symbol": "Ticker", - "timeframe": "Bar Size" - } -} -``` - -```bash -pineforge-backtest \ - --pine strategy.pine \ - --provider csv \ - --provider-config csv.json \ - --venue research \ - --symbol AAPL \ - --timeframe 1m \ - --start 2025-07-01T00:00:00Z \ - --end 2025-07-02T00:00:00Z -``` - -## SQLite - -The native provider safely quotes table and column identifiers after verifying -them against SQLite schema metadata. Tables and views can contain spaces, -reserved words, or otherwise unconventional names. - -```python -from pineforge_data import SqliteBarProvider - -provider = SqliteBarProvider( - "./warehouse.sqlite3", - table="price candles", - venue="warehouse", - mapping={ - "timestamp": "epoch seconds", - "open": "first px", - "high": "top px", - "low": "bottom px", - "close": "last px", - "volume": "traded qty", - "symbol": "security code", - "timeframe": "bar interval", - }, - timestamp_unit="seconds", -) -``` - -Equivalent harness configuration: - -```json -{ - "path": "/data/warehouse.sqlite3", - "table": "price candles", - "timestamp_unit": "seconds", - "columns": { - "timestamp": "epoch seconds", - "open": "first px", - "high": "top px", - "low": "bottom px", - "close": "last px", - "volume": "traded qty", - "symbol": "security code", - "timeframe": "bar interval" - } -} -``` - -Select it with `--provider sqlite`. Numeric timestamp, symbol, and timeframe -filters are pushed into the query; normalization still checks every returned -record. - -## SQLAlchemy-compatible databases - -Install SQLAlchemy support plus the driver for the target database: - -```bash -pip install 'pineforge-data[database]' psycopg -``` - -The provider uses SQLAlchemy Core reflection, so it works with supported -PostgreSQL, MySQL, MariaDB, Oracle, Microsoft SQL Server, SQLite, and -third-party dialects. It intentionally accepts a table or view, not arbitrary -SQL text: - -```python -import os - -from pineforge_data import SqlAlchemyBarProvider - -provider = SqlAlchemyBarProvider( - os.environ["PINEFORGE_DATABASE_URL"], - table="daily_prices", - schema="market_data", - venue="warehouse", - mapping={"timestamp": "trading_day", "symbol": "security_id"}, - timestamp_unit="iso8601", -) -``` - -For joins, expressions, timezone conversion, or vendor-specific data cleanup, -create a database view and map that view. This preserves parameterized filters -and avoids placing raw SQL in a backtest configuration. - -Keep credentials out of JSON by naming an environment variable: - -```json -{ - "url_env": "PINEFORGE_DATABASE_URL", - "schema": "market_data", - "table": "daily_prices", - "timestamp_unit": "iso8601", - "columns": { - "timestamp": "trading_day", - "symbol": "security_id" - } -} -``` - -Select it with `--provider sqlalchemy`. A literal `url` is also accepted for -local or non-sensitive configurations. `engine_options` passes SQLAlchemy -engine options such as pool settings; driver-specific packages remain the -user's dependency. - -Only synchronous SQLAlchemy engines are supported initially. Provider calls -are moved off the asyncio event loop, so they remain compatible with the -async PineForge harness. - -## Symbol and timeframe behavior - -- When a `symbol` column is mapped, catalog listing and exact resolution use - its distinct values, and every fetch filters it. -- Without a `symbol` column, the source is treated as a single-instrument - dataset. `resolve_market("AAPL")` binds the dataset to that requested name. - Set the provider configuration `symbol` when `list_markets()` must advertise - it before resolution. -- When a `timeframe` column is mapped, every request filters it exactly. -- For a single-timeframe dataset without such a column, set configuration - `timeframe` to reject accidental requests at another resolution. - -All sources return rows in ascending time order, apply the half-open interval -`[start_ms, end_ms)`, and apply `limit` after normalization. Duplicate -timestamps for the same requested symbol and timeframe raise -`TabularDataError`; PineForge Data never silently selects one duplicate. - -Local rows are assumed to be confirmed bars because a file or database usually -does not expose a provider clock or candle-close capability. The user owns that -snapshot guarantee. Invalid OHLC relationships, negative volume, missing -values, non-finite numbers, and sub-millisecond timestamp loss fail explicitly. - -## Configuration reference - -| Key | CSV | SQLite | SQLAlchemy | Purpose | -|---|---:|---:|---:|---| -| `path` | required | required | — | Local file/database path | -| `table` | — | required | required | Reflected table or view | -| `url` / `url_env` | — | — | one required | SQLAlchemy connection URL or its environment variable | -| `schema` | — | — | optional | Database schema name | -| `columns` | optional | optional | optional | Partial or complete canonical-to-source mapping | -| `timestamp_unit` | optional | optional | optional | Numeric unit or `iso8601` | -| `timestamp_timezone` | optional | optional | optional | IANA zone for naive datetime values | -| `symbol` | optional | optional | optional | Fixed identity when no symbol column exists | -| `timeframe` | optional | optional | optional | Fixed resolution when no timeframe column exists | -| `encoding`, `delimiter` | optional | — | — | CSV parser settings | -| `engine_options` | — | — | optional | SQLAlchemy `create_engine()` options | - -Unknown configuration keys fail early to catch misspellings. diff --git a/docs/provider-contract.md b/docs/provider-contract.md index 86cdda7..af3884d 100644 --- a/docs/provider-contract.md +++ b/docs/provider-contract.md @@ -88,3 +88,11 @@ passes `--venue` plus the JSON object from `--provider-config` to the factory. - confirmed OHLCV behavior, pagination, deduplication, and source provenance; - close/cleanup behavior and explicit missing-capability errors; - no network access or credentials in CI. + +## Documentation required for a provider PR + +Add `docs/providers/.md` with installation, constructor and registry +configuration, supported capabilities, symbol semantics, examples, errors, and +limitations. Add the provider to the catalog in [providers.md](providers.md). +Provider-specific API details belong on that second-level page; keep the +catalog focused on discovery and shared lifecycle behavior. diff --git a/docs/providers.md b/docs/providers.md index bcc3f3b..d302054 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -1,24 +1,45 @@ -# Using providers +# Provider catalog -Providers are structural Python protocols rather than required base classes. -They are selected independently from venues: +Providers are structural Python implementations selected independently from a +venue and symbol: -- provider: adapter implementation, such as `ccxt`; -- venue: an exchange or broker environment, such as `kraken`; -- symbol: the exact normalized market symbol on that provider. +- **provider** chooses an adapter, such as `ccxt`, `csv`, `sqlite`, or + `sqlalchemy`; +- **venue** identifies an exchange, broker environment, or user-defined data + source, such as `kraken`, `warehouse`, or `research`; +- **symbol** is the exact normalized identity exposed by that provider. -Built-in `csv`, `sqlite`, and `sqlalchemy` providers expose the same catalog and -historical-bar protocol for user-owned data. They discover and map arbitrary -columns at runtime; see [Local files and databases](local-data.md). +This page documents the shared lifecycle and routes to the second-level API +guide for each built-in provider. -## Create a registered provider +## Built-in providers + +| Provider | Data source | Capabilities | Install extra | API guide | +|---|---|---|---|---| +| `ccxt` | Cryptocurrency exchanges | catalog, historical bars, public trades | `ccxt` | [CCXT](providers/ccxt.md) | +| `csv` | Local delimited file | inferred catalog, historical bars | none | [CSV](providers/csv.md) | +| `sqlite` | Local SQLite table or view | reflected catalog, historical bars | none | [SQLite](providers/sqlite.md) | +| `sqlalchemy` | SQLAlchemy-supported database table or view | reflected catalog, historical bars | `database` plus a dialect driver | [SQLAlchemy](providers/sqlalchemy.md) | + +CSV, SQLite, and SQLAlchemy share runtime schema discovery and arbitrary column +mapping. Read the [tabular schema mapping](providers/tabular-schema.md) guide +before configuring one of them. + +## Shared provider lifecycle + +Create a provider by registry name when configuration comes from a CLI, service, +or other runtime boundary: ```python from pineforge_data import create_provider async def resolve_btc_usd(): - provider = create_provider("ccxt", "kraken", config={"enableRateLimit": True}) + provider = create_provider( + "ccxt", + "kraken", + config={"enableRateLimit": True}, + ) try: return await provider.resolve_market("BTC/USD") finally: @@ -26,117 +47,99 @@ async def resolve_btc_usd(): ``` `ProviderRegistry` contains built-in factories and discovers third-party -packages through the `pineforge_data.providers` entry-point group. Unknown or -invalid adapters raise `ProviderNotFoundError` or `ProviderRegistryError`. +packages through the `pineforge_data.providers` entry-point group. Provider +names are case-insensitive; symbols remain exact and provider-defined. -## Discover CCXT markets +Programmatic callers may instantiate a concrete provider directly when they +need constructor options that are not JSON-shaped: ```python -from pineforge_data import CcxtProvider, MarketQuery, MarketType - - -async def list_linear_swaps(): - async with CcxtProvider("okx") as provider: - swaps = await provider.list_markets( - MarketQuery( - market_types=frozenset({MarketType.SWAP}), - quote="USDT", - settle="USDT", - active=True, - linear=True, - ) - ) - for listing in swaps[:10]: - print( - listing.instrument.symbol, - listing.instrument.provider_id, - listing.instrument.contract, - ) -``` - -Use `resolve_market()` before fetching data. Resolution is exact: passing a raw -exchange ID where a unified symbol is expected raises `MarketNotFoundError`. -This prevents an ambiguous ticker from silently choosing spot instead of a -swap, future, or option. +from pineforge_data import CcxtProvider -## Fetch historical bars -```python -from pineforge_data import BarRequest, CcxtProvider - - -async def fetch_swap_bars(): - async with CcxtProvider("okx") as provider: - listing = await provider.resolve_market("BTC/USDT:USDT") - return await provider.fetch_bars( - BarRequest( - instrument=listing.instrument, - timeframe="1h", - start_ms=1_751_328_000_000, - end_ms=1_751_587_200_000, - limit=500, - ) - ) +async def direct_provider(): + async with CcxtProvider("kraken", page_limit=500) as provider: + return await provider.resolve_market("BTC/USD") ``` -CCXT behavior: +## Common market and bar API -- requires the exchange's unified `fetchOHLCV` capability; -- paginates by timestamp up to the requested limit; -- deduplicates repeated candle timestamps; -- sorts results in ascending order; -- excludes bars outside `[start_ms, end_ms)`; -- excludes a candle whose close time is later than the observation time. +Backtest-compatible providers implement `MarketDataProvider`: -Unsupported methods raise `CcxtCapabilityError`; malformed records raise -`CcxtDataError`; missing optional CCXT installation raises -`CcxtDependencyError`. +```python +async def list_markets(query=None): ... +async def resolve_market(symbol): ... +async def fetch_bars(request): ... +async def close(): ... +``` -## Stream public trades +Use exact resolution before fetching bars: ```python -from pineforge_data import CcxtProvider, TradeSubscription +from pineforge_data import BarRequest -async def print_next_trade(): - async with CcxtProvider("kraken") as provider: - listing = await provider.resolve_market("BTC/USD") - subscription = TradeSubscription( +async def fetch(provider, symbol): + listing = await provider.resolve_market(symbol) + return await provider.fetch_bars( + BarRequest( instrument=listing.instrument, + timeframe="1h", start_ms=1_751_328_000_000, - start_sequence=42, + end_ms=1_751_587_200_000, + limit=500, ) - async for tick in provider.stream_trades(subscription): - print(tick.sequence, tick.timestamp_ms, tick.price, tick.quantity) - break + ) ``` -The bootstrap CCXT adapter polls the unified public-trades endpoint, orders each -batch by timestamp, deduplicates by provider trade ID (or normalized values when -no ID exists), and assigns a strictly increasing local sequence. Stop the async -iterator when handing control back to the caller and close the provider. +Every provider returns normalized `MarketListing` and `Bar` records. The end +timestamp is exclusive. Individual provider guides define symbol spelling, +confirmation behavior, pagination, filtering, and source-specific limitations. -## Provider configuration +Live trades are a separate `LiveTradeProvider` capability. A provider that can +fetch historical bars does not automatically promise a live stream. CCXT is +the first built-in provider implementing both. -`CcxtProvider` accepts separate configuration layers: +## Use a provider in the backtest harness -| Argument | Purpose | +The raw-Pine harness accepts every registered `MarketDataProvider`: + +```bash +pineforge-backtest \ + --pine strategy.pine \ + --provider PROVIDER \ + --provider-config provider.json \ + --venue VENUE \ + --symbol SYMBOL \ + --timeframe 1h \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-08T00:00:00Z +``` + +`--provider-config` must contain a JSON object. Its accepted keys belong to the +selected provider and are documented on that provider's API page. Unknown keys +for built-in local/database providers fail early; third-party providers own +their configuration validation. + +## Shared errors + +| Error | Meaning | |---|---| -| `config` | CCXT exchange constructor options and credentials | -| `market_params` | exchange-specific `load_markets` parameters | -| `ohlcv_params` | exchange-specific OHLCV endpoint parameters | -| `trade_params` | exchange-specific public-trade parameters | -| `page_limit` | maximum records requested per REST page | -| `poll_interval_ms` | delay between public-trade polls | -| `dedup_window` | retained trade identity window | +| `ProviderNotFoundError` | no built-in or installed provider has the requested registry name | +| `ProviderRegistryError` | a provider factory is duplicated, invalid, or does not implement `MarketDataProvider` | +| `MarketNotFoundError` | exact symbol resolution failed | +| `SchemaMappingError` | a tabular source cannot be mapped unambiguously to OHLCV | +| `TabularDataError` | a local/database row cannot be normalized safely | -Keep credentials outside source control. The CLI accepts constructor options -through `--provider-config`; programmatic callers can use all endpoint-specific -arguments. +Provider-specific dependency, capability, transport, and malformed-record +errors are listed in each second-level guide. -## Implement another provider +## Add another provider Read the [provider contract](provider-contract.md). Backtest-compatible -providers implement `MarketDataProvider`: market listing, exact resolution, -historical bars, and cleanup. Live trades and macro data remain separate, -optional protocols. +providers implement catalog resolution, historical bars, and cleanup. Live +trades and macro data remain separate optional protocols. + +A provider contribution should also add `docs/providers/.md` and a +row to the catalog above. Keep implementation-specific configuration and +behavior on that second-level page rather than expanding this index. diff --git a/docs/providers/ccxt.md b/docs/providers/ccxt.md new file mode 100644 index 0000000..6d38eb2 --- /dev/null +++ b/docs/providers/ccxt.md @@ -0,0 +1,184 @@ +# CCXT provider API + +[Provider catalog](../providers.md) · [Normalized data model](../data-model.md) + +The `ccxt` provider adapts CCXT's unified async API into PineForge market +listings, confirmed historical OHLCV, and ordered public trade ticks. + +## Install + +```bash +pip install 'pineforge-data[ccxt]' +``` + +The registry name is `ccxt`. The venue is an exact CCXT exchange ID such as +`kraken`, `okx`, or `binance`. + +## Construct the provider + +```python +from pineforge_data import CcxtProvider + +provider = CcxtProvider( + "kraken", + config={"enableRateLimit": True}, + page_limit=500, +) +``` + +`CcxtProvider` is an async context manager and closes exchanges it constructs: + +```python +async with CcxtProvider("kraken") as provider: + listing = await provider.resolve_market("BTC/USD") +``` + +An injected exchange remains caller-owned, which supports tests and shared CCXT +clients. + +## Discover markets + +```python +from pineforge_data import CcxtProvider, MarketQuery, MarketType + + +async def list_linear_swaps(): + async with CcxtProvider("okx") as provider: + return await provider.list_markets( + MarketQuery( + market_types=frozenset({MarketType.SWAP}), + quote="USDT", + settle="USDT", + active=True, + linear=True, + ) + ) +``` + +`Instrument.symbol` uses CCXT's exact unified spelling. `BTC/USDT` and +`BTC/USDT:USDT` are distinct spot and swap listings. `resolve_market()` looks +up the exact unified symbol in the loaded catalog and never infers a market by +parsing text. + +Normalized catalog metadata includes: + +- CCXT market ID as `provider_id`; +- base, quote, and settlement assets; +- spot, swap, future, or option market type; +- active and margin-support flags; +- contract size, linear/inverse flags, expiry, strike, and option side when + supplied by the exchange. + +## Fetch confirmed historical bars + +```python +from pineforge_data import BarRequest, CcxtProvider + + +async def fetch_swap_bars(): + async with CcxtProvider("okx") as provider: + listing = await provider.resolve_market("BTC/USDT:USDT") + return await provider.fetch_bars( + BarRequest( + instrument=listing.instrument, + timeframe="1h", + start_ms=1_751_328_000_000, + end_ms=1_751_587_200_000, + limit=500, + ) + ) +``` + +Historical behavior: + +- requires the exchange's unified `fetchOHLCV` capability; +- translates the requested timeframe with the exchange parser; +- paginates by timestamp up to the requested limit; +- deduplicates repeated candle timestamps; +- sorts bars in ascending order; +- includes only timestamps in `[start_ms, end_ms)`; +- excludes a candle whose close time is later than the observation time. + +Exchange gaps remain gaps. PineForge Data does not synthesize missing candles. + +## Stream public trades + +```python +from pineforge_data import CcxtProvider, TradeSubscription + + +async def print_next_trade(): + async with CcxtProvider("kraken") as provider: + listing = await provider.resolve_market("BTC/USD") + subscription = TradeSubscription( + instrument=listing.instrument, + start_ms=1_751_328_000_000, + start_sequence=42, + ) + async for tick in provider.stream_trades(subscription): + print(tick.sequence, tick.timestamp_ms, tick.price, tick.quantity) + break +``` + +The bootstrap adapter polls the unified public-trades endpoint. It orders each +batch by timestamp, deduplicates by trade ID or normalized values, and assigns +a strictly increasing local sequence. `start_sequence` is the last accepted +sequence, so the next emitted value is `start_sequence + 1`. + +## Constructor reference + +| Argument | Default | Purpose | +|---|---|---| +| `exchange_id` | required | exact CCXT exchange ID and PineForge venue | +| `config` | `{}` | CCXT exchange constructor options and credentials | +| `exchange` | `None` | injected async CCXT-compatible exchange | +| `page_limit` | `1000` | maximum bars or trades requested per REST page | +| `poll_interval_ms` | `1000` | delay between public-trade polls | +| `dedup_window` | `10000` | retained trade identities | +| `reload_markets` | `False` | force CCXT catalog reloads | +| `market_params` | `{}` | exchange-specific `load_markets` parameters | +| `ohlcv_params` | `{}` | exchange-specific OHLCV parameters | +| `trade_params` | `{}` | exchange-specific public-trade parameters | + +Keep credentials out of source control. Constructor configuration may include +API keys for private endpoints, although the built-in bar and trade operations +use public market data. + +## Harness configuration + +The registry factory forwards the JSON object to CCXT's exchange constructor: + +```json +{ + "enableRateLimit": true, + "timeout": 30000 +} +``` + +```bash +pineforge-backtest \ + --pine strategy.pine \ + --provider ccxt \ + --provider-config ccxt.json \ + --venue kraken \ + --symbol BTC/USD \ + --timeframe 15m \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-08T00:00:00Z +``` + +The CLI registry path configures constructor options only. Use the concrete +Python class when endpoint-specific parameters or polling controls are needed. + +## Errors and limitations + +| Error | Meaning | +|---|---| +| `CcxtDependencyError` | the `ccxt` extra is not installed | +| `CcxtCapabilityError` | the exchange lacks `fetchOHLCV` or `fetchTrades` | +| `CcxtDataError` | CCXT returned a record that cannot be normalized safely | +| `MarketNotFoundError` | the exact unified symbol is absent | + +Public trades currently use REST polling, not CCXT Pro WebSockets. Availability, +history depth, pagination semantics, and rate limits still depend on the +selected exchange. diff --git a/docs/providers/csv.md b/docs/providers/csv.md new file mode 100644 index 0000000..e008e6c --- /dev/null +++ b/docs/providers/csv.md @@ -0,0 +1,121 @@ +# CSV provider API + +[Provider catalog](../providers.md) · +[Tabular schema mapping](tabular-schema.md) + +The `csv` provider reads historical OHLCV from a local delimited file. It has +no optional dependency and never modifies the source. + +## Construct the provider + +```python +from pineforge_data import CsvBarProvider + +provider = CsvBarProvider( + "./exports/equity-bars.csv", + venue="research", + mapping={ + "timestamp": "Bucket Start", + "volume": "Total Traded Qty", + }, + timestamp_unit="iso8601", +) +``` + +Plain mappings are partial overrides. Common open, high, low, close, symbol, +and timeframe columns are inferred from the header. Use an explicit +`BarColumnMapping` when every name is custom. + +## Inspect the header + +```python +schema = await provider.inspect_schema() +print(schema.column_names) +mapping = schema.infer_bar_mapping({"timestamp": "Bucket Start"}) +``` + +The first record must be a non-empty, unique header. Files with duplicate or +empty column names fail before data is fetched. + +## Resolve and fetch bars + +```python +from pineforge_data import BarRequest + + +async def load_csv(): + listing = await provider.resolve_market("AAPL") + return await provider.fetch_bars( + BarRequest( + listing.instrument, + timeframe="1m", + start_ms=1_751_328_000_000, + end_ms=1_751_414_400_000, + ) + ) +``` + +When a symbol column is mapped, resolution and fetches use its exact values. +Without one, the complete file is treated as the single symbol passed to +`resolve_market()`. + +The file is scanned for each catalog or bar request. This keeps the provider +dependency-free and deterministic, but SQLite or SQLAlchemy is a better fit for +large datasets queried repeatedly. + +## Constructor reference + +| Argument | Default | Purpose | +|---|---|---| +| `path` | required | local CSV path; `~` is expanded and the path is resolved | +| `venue` | `local` | source identity attached to instruments and provenance | +| `mapping` | inferred | `BarColumnMapping` or partial override mapping | +| `timestamp_unit` | `milliseconds` | numeric timestamp unit or `iso8601` | +| `timestamp_timezone` | `UTC` | IANA zone for naive date/time text | +| `instrument` | `None` | optional fixed instrument template | +| `timeframe` | `None` | optional fixed timeframe assertion | +| `encoding` | `utf-8-sig` | Python text encoding | +| `delimiter` | `,` | exactly one delimiter character | + +## Harness configuration + +```json +{ + "path": "/data/equity-bars.csv", + "encoding": "utf-8-sig", + "delimiter": ",", + "timestamp_unit": "iso8601", + "columns": { + "timestamp": "Bucket Start", + "volume": "Total Traded Qty", + "symbol": "Ticker", + "timeframe": "Bar Size" + } +} +``` + +```bash +pineforge-backtest \ + --pine strategy.pine \ + --provider csv \ + --provider-config csv.json \ + --venue research \ + --symbol AAPL \ + --timeframe 1m \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-02T00:00:00Z +``` + +Registry configuration accepts `path`, `encoding`, `delimiter`, and the shared +[tabular configuration keys](tabular-schema.md#shared-configuration-keys). +Unknown keys fail early. + +## Errors and limitations + +- A missing path raises `FileNotFoundError`. +- Missing, duplicate, or ambiguous columns raise `SchemaMappingError`. +- Rows with more values than the header or invalid OHLCV raise + `TabularDataError`. +- CSV parsing is synchronous work moved off the asyncio event loop. +- No dialect sniffing is performed; specify `delimiter` explicitly. +- The provider does not cache an index or file contents between requests. diff --git a/docs/providers/sqlalchemy.md b/docs/providers/sqlalchemy.md new file mode 100644 index 0000000..dfb84ec --- /dev/null +++ b/docs/providers/sqlalchemy.md @@ -0,0 +1,161 @@ +# SQLAlchemy provider API + +[Provider catalog](../providers.md) · +[Tabular schema mapping](tabular-schema.md) + +The `sqlalchemy` provider reflects one table or view through a synchronous +SQLAlchemy 2.x engine. It supports SQLAlchemy built-in and third-party dialects +without requiring ORM models or a PineForge-owned DDL. + +## Install + +Install PineForge's database extra and the driver for the target dialect: + +```bash +pip install 'pineforge-data[database]' psycopg +``` + +The `database` extra installs SQLAlchemy, not every database driver. For +example, PostgreSQL may use `psycopg`, MySQL may use `pymysql`, and an +organization-specific dialect may have its own package. + +## Construct the provider + +```python +import os + +from pineforge_data import SqlAlchemyBarProvider + +provider = SqlAlchemyBarProvider( + os.environ["PINEFORGE_DATABASE_URL"], + table="daily_prices", + schema="market_data", + venue="warehouse", + mapping={ + "timestamp": "trading_day", + "symbol": "security_id", + }, + timestamp_unit="iso8601", +) +``` + +The URL must select a synchronous SQLAlchemy driver. Provider calls run in a +worker thread so they do not block the async PineForge harness. + +Use a database role with `SELECT` access only. PineForge Data issues reflection +and select operations, but database permissions remain the strongest safety +boundary. + +## Inspect reflected metadata + +```python +schema = await provider.inspect_schema() +for column in schema.columns: + print(column.name, column.data_type, column.nullable) +``` + +Reflection loads the exact table/view and optional schema. The returned source +identity includes the PineForge venue and qualified table, not the connection +URL or credentials. + +## Resolve and fetch bars + +```python +from pineforge_data import BarRequest + + +async def load_database(): + listing = await provider.resolve_market("AAPL") + return await provider.fetch_bars( + BarRequest( + listing.instrument, + timeframe="1d", + start_ms=1_735_689_600_000, + end_ms=1_751_414_400_000, + ) + ) +``` + +Mapped symbol and timeframe columns use SQLAlchemy bound expressions. Numeric +timestamp ranges are converted into the configured source unit and pushed into +the query. Every row is checked again by the shared normalization layer. + +For joins, expressions, timezone conversion, or vendor-specific cleanup, +create a database view and map the view. The provider intentionally does not +accept arbitrary SQL strings in harness configuration. + +## Constructor reference + +| Argument | Default | Purpose | +|---|---|---| +| `url` | required | synchronous SQLAlchemy database URL | +| `table` | required | exact reflected table or view | +| `venue` | `database` | source identity attached to instruments and provenance | +| `schema` | `None` | optional database schema | +| `mapping` | inferred | `BarColumnMapping` or partial overrides | +| `timestamp_unit` | `milliseconds` | numeric timestamp unit or `iso8601` | +| `timestamp_timezone` | `UTC` | IANA zone for naive date/datetime values | +| `instrument` | `None` | optional fixed instrument template | +| `timeframe` | `None` | optional fixed timeframe assertion | +| `engine_options` | `{}` | keyword options forwarded to `create_engine()` | + +`hide_parameters=True` is enabled unless explicitly overridden in +`engine_options` so SQL logs do not include bound symbol or time values. + +Call `await provider.close()` to dispose the engine and its connection pool. + +## Harness configuration + +Keep credentials out of JSON by naming an environment variable: + +```json +{ + "url_env": "PINEFORGE_DATABASE_URL", + "schema": "market_data", + "table": "daily_prices", + "timestamp_unit": "iso8601", + "columns": { + "timestamp": "trading_day", + "symbol": "security_id" + }, + "engine_options": { + "pool_pre_ping": true, + "pool_recycle": 1800 + } +} +``` + +```bash +export PINEFORGE_DATABASE_URL='postgresql+psycopg://user:password@db/market' +pineforge-backtest \ + --pine strategy.pine \ + --provider sqlalchemy \ + --provider-config database.json \ + --venue warehouse \ + --symbol AAPL \ + --timeframe 1d \ + --start 2025-01-01T00:00:00Z \ + --end 2025-07-02T00:00:00Z +``` + +A literal `url` is also accepted for local or non-sensitive configurations. +Configure only one of `url` and `url_env`. Registry configuration also accepts +`table`, `schema`, `engine_options`, and the shared +[tabular configuration keys](tabular-schema.md#shared-configuration-keys). + +## Errors and limitations + +| Error | Meaning | +|---|---| +| `SqlAlchemyDependencyError` | the `database` extra is not installed | +| `SchemaMappingError` | reflected columns cannot satisfy the OHLCV mapping | +| `TabularDataError` | a selected row cannot be normalized safely | + +Driver import, authentication, reflection, connectivity, and database timeout +errors remain SQLAlchemy/dialect exceptions so operational tooling retains the +original cause. + +Only synchronous engines are supported initially. Each provider owns one +engine and reflected table cache. Query pagination is delegated to the database +filter and ordering plan; PineForge Data currently materializes the selected +rows before final normalization and `limit` application. diff --git a/docs/providers/sqlite.md b/docs/providers/sqlite.md new file mode 100644 index 0000000..02899a3 --- /dev/null +++ b/docs/providers/sqlite.md @@ -0,0 +1,131 @@ +# SQLite provider API + +[Provider catalog](../providers.md) · +[Tabular schema mapping](tabular-schema.md) + +The `sqlite` provider reflects one local SQLite table or view and queries +historical OHLCV through Python's built-in SQLite driver. It requires no +optional package. + +## Construct the provider + +```python +from pineforge_data import SqliteBarProvider + +provider = SqliteBarProvider( + "./warehouse.sqlite3", + table="price candles", + venue="warehouse", + mapping={ + "timestamp": "epoch seconds", + "open": "first px", + "high": "top px", + "low": "bottom px", + "close": "last px", + "volume": "traded qty", + "symbol": "security code", + "timeframe": "bar interval", + }, + timestamp_unit="seconds", +) +``` + +Table and column names may contain spaces or reserved words. The provider first +checks the table or view and mapping against `sqlite_schema` and `PRAGMA +table_info`, then quotes the reflected identifiers. + +## Inspect the table or view + +```python +schema = await provider.inspect_schema() +for column in schema.columns: + print(column.name, column.data_type, column.nullable) +``` + +The provider opens a fresh read-only SQLite connection for each operation. It +never creates tables, runs migrations, or changes source data. + +## Resolve and fetch bars + +```python +from pineforge_data import BarRequest + + +async def load_sqlite(): + listing = await provider.resolve_market("AAPL") + return await provider.fetch_bars( + BarRequest( + listing.instrument, + timeframe="1m", + start_ms=1_751_328_000_000, + end_ms=1_751_414_400_000, + ) + ) +``` + +Mapped symbol and timeframe values are bound parameters. Numeric timestamp +bounds are translated into the configured source unit and pushed into the SQL +query. The common normalization layer checks the interval again after reading +each row. + +ISO-8601 timestamps are filtered after conversion because arbitrary stored text +formats do not always preserve chronological SQL ordering. For large text-time +tables, expose a view with a consistently sortable numeric epoch column. + +## Constructor reference + +| Argument | Default | Purpose | +|---|---|---| +| `path` | required | local SQLite database path | +| `table` | required | exact table or view name | +| `venue` | `local` | source identity attached to instruments and provenance | +| `mapping` | inferred | `BarColumnMapping` or partial overrides | +| `timestamp_unit` | `milliseconds` | numeric timestamp unit or `iso8601` | +| `timestamp_timezone` | `UTC` | IANA zone for naive date/time values | +| `instrument` | `None` | optional fixed instrument template | +| `timeframe` | `None` | optional fixed timeframe assertion | + +## Harness configuration + +```json +{ + "path": "/data/warehouse.sqlite3", + "table": "price candles", + "timestamp_unit": "seconds", + "columns": { + "timestamp": "epoch seconds", + "open": "first px", + "high": "top px", + "low": "bottom px", + "close": "last px", + "volume": "traded qty", + "symbol": "security code", + "timeframe": "bar interval" + } +} +``` + +```bash +pineforge-backtest \ + --pine strategy.pine \ + --provider sqlite \ + --provider-config sqlite.json \ + --venue warehouse \ + --symbol AAPL \ + --timeframe 1m \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-02T00:00:00Z +``` + +Registry configuration accepts `path`, `table`, and the shared +[tabular configuration keys](tabular-schema.md#shared-configuration-keys). +Unknown keys fail early. + +## Errors and limitations + +- A missing database path raises `FileNotFoundError`. +- A missing table/view or a source with no columns raises `TabularDataError`. +- Missing or ambiguous mapped fields raise `SchemaMappingError`. +- SQLite connections use URI `mode=ro`; the filesystem must permit reads. +- Complex joins, expressions, or cleanup should be exposed as a SQLite view. +- The provider uses synchronous SQLite calls moved off the asyncio event loop. diff --git a/docs/providers/tabular-schema.md b/docs/providers/tabular-schema.md new file mode 100644 index 0000000..e11c082 --- /dev/null +++ b/docs/providers/tabular-schema.md @@ -0,0 +1,152 @@ +# Tabular schema mapping + +[Provider catalog](../providers.md) · [CSV API](csv.md) · +[SQLite API](sqlite.md) · [SQLAlchemy API](sqlalchemy.md) + +CSV, SQLite, and SQLAlchemy providers normalize user-owned OHLCV without +requiring a PineForge DDL. They share one runtime discovery and mapping API: + +```text +file / table / view + ↓ inspect header or reflected metadata +timestamp, open, high, low, close, volume, [symbol], [timeframe] + ↓ validate and normalize +PineForge Bar records +``` + +## Inspect a source + +Every tabular provider exposes `inspect_schema()` before rows are normalized: + +```python +from pineforge_data import CsvBarProvider + + +async def inspect_file(): + provider = CsvBarProvider("./vendor-export.csv", venue="research") + schema = await provider.inspect_schema() + for column in schema.columns: + print(column.name, column.data_type, column.nullable) +``` + +`TabularSchema` contains the source identity and ordered `SourceColumn` +records. CSV types are reported as text; database providers report reflected +types and nullability. + +## Infer common columns + +`TabularSchema.infer_bar_mapping()` recognizes common names such as +`timestamp`, `datetime`, `open_time`, `PX_OPEN`, `qty`, `ticker`, and +`interval`. Supply only unusual fields: + +```python +mapping = schema.infer_bar_mapping( + { + "timestamp": "Bucket Start", + "volume": "Total Traded Qty", + } +) +``` + +Inference never chooses between multiple plausible columns. Missing or +ambiguous fields raise `SchemaMappingError` containing the available columns +and the overrides needed to continue. + +Provider constructors also accept a plain mapping. A plain mapping is a set of +partial overrides; omitted fields are inferred when the source is inspected. + +## Define an explicit mapping + +For a fully custom schema, map all six required OHLCV fields: + +```python +from pineforge_data import BarColumnMapping + +mapping = BarColumnMapping( + timestamp="epoch seconds", + open="first px", + high="top px", + low="bottom px", + close="last px", + volume="traded qty", + symbol="security code", # optional + timeframe="bar interval", # optional +) +``` + +One source column cannot provide multiple canonical fields. Every selected +column must exist exactly in the inspected source. + +## Timestamp values + +Normalized timestamps are Unix milliseconds. Set `timestamp_unit` for numeric +source values: + +| Value | Meaning | +|---|---| +| `seconds` or `s` | Unix seconds, including exact fractional milliseconds | +| `milliseconds` or `ms` | Unix milliseconds; the default | +| `microseconds` or `us` | Unix microseconds | +| `nanoseconds` or `ns` | Unix nanoseconds | +| `iso8601` | ISO-8601 text or database date/datetime objects | + +ISO-8601 values with an offset or `Z` are unambiguous. Naive text or database +datetimes use `timestamp_timezone`, which defaults to `UTC` and accepts an IANA +zone such as `America/New_York`. Values that cannot be represented exactly at +millisecond precision fail rather than being rounded silently. + +## Symbol behavior + +- When a `symbol` column is mapped, `list_markets()` uses its distinct values, + `resolve_market()` requires an exact value, and bar fetches filter it. +- Without a `symbol` column, the source is treated as a single-instrument + dataset. `resolve_market("AAPL")` binds that dataset to the requested name. +- Set the provider configuration `symbol` when a single-instrument dataset + should be advertised by `list_markets()` before resolution. + +Symbols are compared exactly. A provider never rewrites case, punctuation, or +contract suffixes. + +## Timeframe behavior + +- When a `timeframe` column is mapped, each fetch filters it exactly. +- For a single-timeframe source without that column, set configuration + `timeframe` to reject requests at another resolution. +- When neither exists, the request timeframe is an assertion supplied by the + caller; PineForge Data cannot verify the source resolution. + +## Shared normalization guarantees + +Tabular providers: + +- include only bars in the half-open interval `[start_ms, end_ms)`; +- sort results in ascending timestamp order; +- apply `limit` after normalization; +- reject duplicate timestamps for the requested symbol and timeframe; +- reject missing or non-finite values, invalid OHLC relationships, negative + volume, and lossy sub-millisecond timestamps; +- attach the requested instrument and provider source to every `Bar`. + +Rows are assumed to be confirmed snapshots because local files and databases +usually do not expose a provider clock or candle-close capability. The user +owns that snapshot guarantee. + +## Shared configuration keys + +| Key | Default | Purpose | +|---|---|---| +| `columns` | inferred | partial canonical-to-source mapping | +| `timestamp_unit` | `milliseconds` | numeric unit or `iso8601` | +| `timestamp_timezone` | `UTC` | IANA zone for naive date/datetime values | +| `symbol` | unset | fixed identity when no symbol column exists | +| `timeframe` | unset | fixed resolution when no timeframe column exists | + +Provider-specific path, table, URL, parser, and engine keys are documented on +the individual API pages. + +## Errors + +`SchemaMappingError` reports a discovery or mapping problem before bar +normalization. `TabularDataError` reports a malformed row, duplicate timestamp, +or unsafe conversion. Both include source-facing context while avoiding +database credentials. diff --git a/tests/test_documentation.py b/tests/test_documentation.py index eb7bba6..fb685a1 100644 --- a/tests/test_documentation.py +++ b/tests/test_documentation.py @@ -8,10 +8,18 @@ ROOT = Path(__file__).resolve().parents[1] MARKDOWN_LINK = re.compile(r"\[[^]]*]\(([^)]+)\)") -DOCUMENTS = [ROOT / "README.md", ROOT / "CONTRIBUTING.md", *sorted((ROOT / "docs").glob("*.md"))] +DOCUMENTS = [ + ROOT / "README.md", + ROOT / "CONTRIBUTING.md", + *sorted((ROOT / "docs").rglob("*.md")), +] -@pytest.mark.parametrize("document", DOCUMENTS, ids=lambda path: path.name) +@pytest.mark.parametrize( + "document", + DOCUMENTS, + ids=lambda path: str(path.relative_to(ROOT)), +) def test_relative_documentation_links_exist(document: Path) -> None: for raw_target in MARKDOWN_LINK.findall(document.read_text(encoding="utf-8")): target = raw_target.split("#", 1)[0]