Abstract provider market catalogs

CONTRIBUTING.md

@@ -1,7 +1,8 @@
 # Contributing
 
 Community data providers are the reason this repository exists. Keep the core
-contracts small and put vendor behavior behind provider modules.
+contracts small and put vendor behavior behind Python provider modules. This
+repository does not accept a separate C++ provider implementation surface.
 
 Docker and initialized codegen/engine submodules are required for raw-Pine
 integration work:
@@ -13,7 +14,14 @@ docker version
 
 ## Provider checklist
 
-- Implement one or more protocols from `pineforge_data.providers`.
+- Implement `MarketDataProvider` for backtest-compatible adapters; live-trade
+  and macro support remain separate optional protocols.
+- Resolve exact normalized symbols from the upstream market catalog. Do not
+  infer base, quote, settlement currency, or contract terms by parsing symbols.
+- Preserve both the normalized symbol and the provider-native `provider_id`.
+- Represent spot/cash/CFD/swap/future/option separately and populate derivative
+  contract size, linear/inverse flags, expiry, strike, and option type when the
+  upstream supplies them.
 - Normalize all timestamps to Unix milliseconds.
 - Preserve the source name and normalized instrument on every record.
 - Keep credentials out of logs and exception messages; prefer authorization
@@ -25,6 +33,18 @@ docker version
   extending the normalized records.
 - Never make `pineforge-engine` depend on this package.
 
+Adapters may be contributed in-tree or shipped by another Python package. An
+external package registers a factory without changing this repository's CLI:
+
+```toml
+[project.entry-points."pineforge_data.providers"]
+example = "example_pineforge:provider_factory"
+```
+
+The callable receives `(venue, config)` and returns a structural
+`MarketDataProvider`. See [docs/provider-contract.md](docs/provider-contract.md)
+for the normalized model and a complete skeleton.
+
 ## Determinism and macro vintages
 
 Historical results must be reproducible. Cache or snapshot mutable upstream

README.md

@@ -21,18 +21,19 @@ and asynchronous I/O. Python makes those integrations accessible to community
 contributors. Engine throughput remains native: normalized records are packed
 into contiguous C ABI arrays and submitted in one call.
 
-If profiling later identifies a normalization hot path, it can gain an optional
-native extension without changing the public provider contracts.
-
 ## Initial contracts
 
-- `Instrument` — normalized symbol, venue, timezone, session, and volume units.
+- `Instrument` — normalized symbol, provider-native ID, venue, asset/market type,
+  currencies, contract terms, timezone, session, and volume units.
+- `MarketListing` and `MarketQuery` — catalog discovery without vendor schemas.
 - `Bar` — confirmed OHLCV with source provenance.
 - `TradeTick` — the provider-neutral four-field engine payload plus provenance.
 - `MacroObservation` — observation period, first release, and vintage timestamps
   to prevent revised-data lookahead.
-- `HistoricalBarProvider`, `LiveTradeProvider`, and `MacroDataProvider` — small
-  structural protocols that community adapters implement.
+- `MarketCatalogProvider`, `HistoricalBarProvider`, `LiveTradeProvider`, and
+  `MacroDataProvider` — small structural protocols that community adapters
+  implement.
+- `ProviderRegistry` — built-in and installed broker adapters selected by name.
 - `PfBar`, `PfTradeTick`, and `EngineStreamSink` — dependency-free `ctypes`
   interoperability with PineForge strategy libraries.
 
@@ -59,14 +60,21 @@ request = BarRequest(
 )
 
 async with CcxtProvider("kraken") as provider:
+    listing = await provider.resolve_market("BTC/USDT")
     confirmed_bars = await provider.fetch_bars(request)
 ```
 
-`Instrument.symbol` uses CCXT's unified spelling. Exchange credentials and
-exchange-specific options can be passed through `config`, while endpoint
-options remain isolated in `ohlcv_params` and `trade_params`. Realtime public
-trades use REST polling in this bootstrap; a WebSocket transport can implement
-the same `LiveTradeProvider` contract later.
+`Instrument.symbol` uses CCXT's exact unified spelling. Do not infer a market
+by parsing it: `resolve_market()` uses CCXT's catalog fields to distinguish
+spot, swap, future, and option listings and captures `base`, `quote`, `settle`,
+raw exchange ID, contract size, linear/inverse settlement, expiry, strike, and
+option type. For example, `BTC/USDT` and `BTC/USDT:USDT` are separate markets.
+
+Exchange credentials and exchange-specific options can be passed through
+`config`, while endpoint options remain isolated in `market_params`,
+`ohlcv_params`, and `trade_params`. Realtime public trades use REST polling in
+this bootstrap; a WebSocket transport can implement the same
+`LiveTradeProvider` contract later.
 
 `TradeSubscription.start_ms` can pin the live handoff to the next timestamp
 after an engine warmup. `start_sequence` is the last accepted sequence, so the
@@ -104,7 +112,8 @@ git submodule update --init
 ```bash
 .venv/bin/pineforge-backtest \
   --pine strategy.pine \
-  --exchange kraken \
+  --provider ccxt \
+  --venue kraken \
   --symbol BTC/USD \
   --timeframe 15m \
   --start 2026-07-01T00:00:00Z \
@@ -136,11 +145,9 @@ source-available under its own PolyForm Noncommercial license and supplemental
 terms; review `vendor/pineforge-codegen-oss/LICENSE` before distribution or
 commercial use. The engine remains Apache-2.0.
 
-Provider implementations are organized by their strongest supported runtime.
-The current Python bucket contains CCXT and the harness; native low-latency
-providers live under `cpp/providers`. Both buckets must emit the same normalized
-records, but an individual provider does not need implementations in both
-languages.
+Provider implementations in this repository are Python-only. The compiled C++
+strategy and engine stay behind the Docker/runtime boundary; broker SDKs and
+provider-specific types do not cross into `pineforge-engine`.
 
 ## Development
 
@@ -157,14 +164,16 @@ PINEFORGE_DOCKER_TEST=1 .venv/bin/pytest tests/test_docker_integration.py
 
 A provider adapter should:
 
-1. fetch or subscribe to its external service;
-2. retain source and instrument provenance;
-3. normalize timestamps to Unix milliseconds and records to the public models;
-4. emit stable ordering sequences when the provider supplies them;
-5. batch records before crossing the engine ABI when practical.
+1. expose exact market discovery and symbol resolution;
+2. fetch or subscribe to its external service;
+3. retain source and instrument provenance;
+4. normalize timestamps to Unix milliseconds and records to the public models;
+5. emit stable ordering sequences when the provider supplies them;
+6. batch records before crossing the engine ABI when practical.
 
 It should not add provider-specific fields to `pineforge-engine`. Data that the
 engine does not consume remains in provider-owned metadata or higher-level
 models in this repository.
 
-See [CONTRIBUTING.md](CONTRIBUTING.md) before adding a provider.
+See [the provider contract](docs/provider-contract.md) and
+[CONTRIBUTING.md](CONTRIBUTING.md) before adding a provider.

docs/provider-contract.md

@@ -0,0 +1,90 @@
+# Provider contract
+
+PineForge Data treats a provider and a venue as separate identities:
+
+- **provider** selects an adapter implementation, such as `ccxt` or a broker SDK;
+- **venue** selects the exchange, broker environment, or data source instance;
+- **symbol** is the provider's normalized public symbol;
+- **provider_id** is the raw venue identifier used for provenance and debugging.
+
+An adapter must resolve an exact symbol through its upstream catalog before it
+fetches bars. Symbol text is not a portable schema: derivative symbols can
+encode settlement currency, expiry, strike, or option side differently across
+providers.
+
+## Normalized market model
+
+`Instrument` carries:
+
+- `asset_class`: crypto, equity, forex, commodity, index, fund, bond, or unknown;
+- `market_type`: spot, cash, swap, future, option, CFD, or unknown;
+- `base`, `quote`, and `settle` currencies/assets;
+- `contract`: optional `ContractSpec` with contract size, linear/inverse flags,
+  expiry, strike, and option type;
+- venue, normalized symbol, provider ID, volume unit, timezone, and session.
+
+Margin availability is a capability on `MarketListing`, not a market type. A
+spot listing can support margin while remaining a spot market.
+
+Unknown and unavailable fields should remain explicit (`UNKNOWN`, empty text,
+or `None`). An adapter must not manufacture metadata by parsing the symbol.
+
+## In-tree adapter skeleton
+
+```python
+from collections.abc import Mapping, Sequence
+
+from pineforge_data import (
+    Bar,
+    BarRequest,
+    Instrument,
+    MarketListing,
+    MarketNotFoundError,
+    MarketQuery,
+)
+
+
+class BrokerProvider:
+    def __init__(self, venue: str, config: Mapping[str, object]) -> None:
+        self.venue = venue
+        self.name = f"broker:{venue}"
+
+    async def list_markets(
+        self, query: MarketQuery | None = None
+    ) -> Sequence[MarketListing]:
+        listings = []  # Normalize the broker's catalog here.
+        return listings if query is None else [x for x in listings if query.matches(x)]
+
+    async def resolve_market(self, symbol: str) -> MarketListing:
+        for listing in await self.list_markets():
+            if listing.instrument.symbol == symbol:
+                return listing
+        raise MarketNotFoundError(f"no exact market {symbol!r} on {self.venue}")
+
+    async def fetch_bars(self, request: BarRequest) -> Sequence[Bar]:
+        ...
+
+    async def close(self) -> None:
+        ...
+
+
+def provider_factory(
+    venue: str, config: Mapping[str, object]
+) -> BrokerProvider:
+    return BrokerProvider(venue, config)
+```
+
+Add an in-tree factory to `providers/registry.py`. Out-of-tree packages use the
+`pineforge_data.providers` Python entry-point group shown in
+`CONTRIBUTING.md`. The CLI accepts any registered name through `--provider` and
+passes `--venue` plus the JSON object from `--provider-config` to the factory.
+
+## Tests required for a provider PR
+
+- offline catalog fixtures for every supported market type;
+- exact symbol resolution and a missing-symbol error;
+- normalized provider ID, venue, base/quote/settle, and contract terms;
+- catalog filtering through `MarketQuery`;
+- confirmed OHLCV behavior, pagination, deduplication, and source provenance;
+- close/cleanup behavior and explicit missing-capability errors;
+- no network access or credentials in CI.

src/pineforge_data/__init__.py

@@ -15,7 +15,17 @@
 )
 from .engine import EngineStreamSink, PfBar, PfTradeTick, pack_bars, pack_trade_ticks
 from .errors import EngineStreamError
-from .models import Bar, Instrument, MacroObservation, TradeTick
+from .models import (
+    AssetClass,
+    Bar,
+    ContractSpec,
+    Instrument,
+    MacroObservation,
+    MarketListing,
+    MarketType,
+    OptionType,
+    TradeTick,
+)
 from .providers import (
     CcxtCapabilityError,
     CcxtDataError,
@@ -25,10 +35,20 @@
     HistoricalBarProvider,
     LiveTradeProvider,
     MacroDataProvider,
+    MarketCatalogProvider,
+    MarketDataProvider,
+    MarketNotFoundError,
+    ProviderFactory,
+    ProviderNotFoundError,
+    ProviderRegistry,
+    ProviderRegistryError,
+    create_provider,
+    default_registry,
 )
-from .requests import BarRequest, MacroRequest, TradeSubscription
+from .requests import BarRequest, MacroRequest, MarketQuery, TradeSubscription
 
 __all__ = [
+    "AssetClass",
     "BacktestOptions",
     "BacktestReport",
     "Bar",
@@ -38,6 +58,7 @@
     "CcxtDependencyError",
     "CcxtError",
     "CcxtProvider",
+    "ContractSpec",
     "DockerBacktestRuntime",
     "DockerExecutionError",
     "DockerPrerequisiteError",
@@ -51,11 +72,24 @@
     "MacroObservation",
     "MacroRequest",
     "MagnifierDistribution",
+    "MarketCatalogProvider",
+    "MarketDataProvider",
+    "MarketListing",
+    "MarketNotFoundError",
+    "MarketQuery",
+    "MarketType",
+    "OptionType",
     "PfBar",
     "PfTradeTick",
     "PineForgeBacktestRunner",
+    "ProviderFactory",
+    "ProviderNotFoundError",
+    "ProviderRegistry",
+    "ProviderRegistryError",
     "TradeSubscription",
     "TradeTick",
+    "create_provider",
+    "default_registry",
     "discover_repository_root",
     "pack_bars",
     "pack_trade_ticks",