diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a933585..d29c567 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,6 +4,11 @@ Community data providers are the reason this repository exists. Keep the core contracts small and put vendor behavior behind Python provider modules. This repository does not accept a separate C++ provider implementation surface. +User-facing architecture and API guides are indexed in +[docs/index.md](docs/index.md). Update the applicable guide when changing public +models, provider behavior, CLI options, report fields, runtime channels, or +server/cache semantics. + Docker is required for raw-Pine integration work. Engine and codegen are consumed only through the pinned `pineforge-release` image; do not add source submodules or duplicate their build logic here. diff --git a/README.md b/README.md index 4fedbdc..372e13b 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,22 @@ The engine does not import or link this package. Provider transport, authentication, retries, caching, symbol mapping, and vendor schemas stay here; the engine receives only normalized bars and ordered trades. +## Documentation + +- [Documentation home](docs/index.md) — architecture, guarantees, and guide map. +- [Getting started](docs/getting-started.md) — installation, first provider + request, and local or remote backtest. +- [Normalized data model](docs/data-model.md) — instruments, contracts, bars, + live trades, macro vintages, and validation rules. +- [Using providers](docs/providers.md) — registry, CCXT market discovery, + historical bars, live trades, configuration, and errors. +- [Backtesting](docs/backtesting.md) — CLI options, configuration files, runtime + channels, report schema, and reproducibility. +- [FastAPI server](docs/server.md) — concurrency, authentication, timeouts, + compile cache, and deployment. +- [Provider contract](docs/provider-contract.md) — implementing and testing a + community exchange or broker adapter. + ## Why Python first Provider integrations are dominated by HTTP, WebSockets, JSON, credentials, @@ -189,31 +205,69 @@ hit/key/hash are included in response provenance. See [docs/server.md](docs/server.md) for endpoints, limits, deployment, and cache settings. -## Development +## Contributing -```bash -python3 -m venv .venv -.venv/bin/pip install -e '.[dev,ccxt,server]' -.venv/bin/ruff check . -.venv/bin/mypy src -.venv/bin/pytest -PINEFORGE_DOCKER_TEST=1 .venv/bin/pytest tests/test_docker_integration.py -``` +Community providers, market-model improvements, server/runtime work, tests, and +documentation are welcome. Provider integrations are Python-only; engine and +codegen changes belong in their upstream repositories and are consumed here +through `pineforge-release`. -## Provider boundary +### Choose the right contribution path -A provider adapter should: +| Contribution | Primary location | Start here | +|---|---|---| +| Exchange or broker adapter | `src/pineforge_data/providers/` | [Provider contract](docs/provider-contract.md) | +| Market, contract, bar, or request model | `src/pineforge_data/models.py`, `src/pineforge_data/requests.py`, `src/pineforge_data/providers/base.py` | Existing public models and protocols | +| Backtest harness or HTTP client | `src/pineforge_data/cli/backtest.py`, `src/pineforge_data/server_client.py` | Harness unit tests | +| FastAPI concurrency or compile cache | `src/pineforge_data/server.py`, `src/pineforge_data/compile_cache.py` | [Server guide](docs/server.md) | +| Release-container integration | `src/pineforge_data/release_contract.py`, `src/pineforge_data/docker_runtime.py` | Pinned release contract and Docker tests | +| Documentation or examples | `README.md`, `docs/` | A focused documentation PR | -1. expose exact market discovery and symbol resolution; -2. fetch or subscribe to its external service; -3. retain source and instrument provenance; -4. normalize timestamps to Unix milliseconds and records to the public models; -5. emit stable ordering sequences when the provider supplies them; -6. batch records before crossing the engine ABI when practical. +For a new provider, implement the smallest applicable structural protocols, +register its factory, keep its SDK in an optional dependency extra, and add +offline fixture tests. Resolve exact upstream markets through their catalog; +do not parse symbols to infer base, quote, settlement, or contract terms. +Provider-specific fields stay in this repository and must not leak into +`pineforge-engine`. -It should not add provider-specific fields to `pineforge-engine`. Data that the -engine does not consume remains in provider-owned metadata or higher-level -models in this repository. +### Development setup + +```bash +git clone https://github.com/pineforge-4pass/pineforge-data.git +cd pineforge-data +python3 -m venv .venv +.venv/bin/pip install -e '.[dev,ccxt,server]' +``` -See [the provider contract](docs/provider-contract.md) and -[CONTRIBUTING.md](CONTRIBUTING.md) before adding a provider. +No Git submodules are required. Docker is needed only for release-runtime and +end-to-end backtest work. + +### Before opening a pull request + +1. Keep the change focused and document any public API, report-schema, provider, + runtime-image, or cache-key compatibility impact. +2. Add deterministic offline tests; CI must not require credentials or live + provider access. +3. Keep credentials out of fixtures, logs, exception messages, and committed + configuration. +4. Run the standard checks: + + ```bash + .venv/bin/ruff format --check src tests + .venv/bin/ruff check . + .venv/bin/mypy src + .venv/bin/pytest + .venv/bin/python -m build + ``` + +5. For Docker, FastAPI server, cache, or release-contract changes, also run: + + ```bash + PINEFORGE_DOCKER_TEST=1 .venv/bin/pytest tests/test_docker_integration.py + ``` + +Read the [documentation home](docs/index.md) and +[CONTRIBUTING.md](CONTRIBUTING.md) for provider requirements, +determinism rules, external provider entry points, and the complete checklist. +For broad changes to public models or the report contract, open an issue first +so providers and runtime consumers can agree on the shape before implementation. diff --git a/docs/backtesting.md b/docs/backtesting.md new file mode 100644 index 0000000..3bf9b7c --- /dev/null +++ b/docs/backtesting.md @@ -0,0 +1,172 @@ +# Backtesting with PineForge Data + +The harness accepts raw PineScript. Data acquisition stays in Python while +transpilation, compilation, and deterministic execution stay in +`pineforge-release`. + +## Execution modes + +### Local release container + +Without a server URL, the harness pulls and runs the pinned release image +locally. The container has networking disabled, a read-only root filesystem, +dropped capabilities, `no-new-privileges`, a read-only input mount, and an +executable `/tmp` tmpfs required for strategy compilation. + +### FastAPI server + +Set `--server-url` or `PINEFORGE_SERVER_URL` to send raw Pine and normalized +bars to the concurrent server. Provider credentials and provider API calls stay +on the harness host. Set `PINEFORGE_SERVER_API_KEY` for bearer authentication. + +The server returns synchronously while accepting multiple requests. Capacity, +queueing, timeouts, authentication, deployment, and compiled-strategy caching +are documented in the [server guide](server.md). + +## CLI reference + +```bash +pineforge-backtest \ + --pine strategy.pine \ + --provider ccxt \ + --venue kraken \ + --symbol BTC/USD \ + --timeframe 15m \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-08T00:00:00Z +``` + +| Group | Options | +|---|---| +| Strategy | `--pine`, `--strategy-params`, `--strategy-overrides` | +| Data source | `--provider`, `--venue`/`--exchange`, `--symbol`, `--timeframe`, `--start`, `--end`, `--limit`, `--provider-config` | +| Pine context | `--timezone`, `--session`, `--engine-timeframe`, `--script-timeframe` | +| Fill modeling | `--bar-magnifier`, `--magnifier-samples` | +| Local runtime | `--runtime-image`/`--image`, `--pull-policy`, `--execution-timeout` | +| Remote runtime | `--server-url`, `--server-api-key-env`, `--execution-timeout` | +| Output | `--output`, `--pretty` | + +`--start` and `--end` accept Unix milliseconds or timezone-aware ISO-8601. The +end is exclusive. `--engine-timeframe` defaults to a Pine-compatible conversion +of the provider timeframe, and `--script-timeframe` defaults to the engine +timeframe. + +## Configuration files + +Provider constructor configuration: + +```json +{ + "enableRateLimit": true, + "timeout": 30000 +} +``` + +```bash +--provider-config provider.json +``` + +Pine input overrides: + +```json +{ + "Fast Length": 8, + "Slow Length": 21 +} +``` + +```bash +--strategy-params inputs.json +``` + +`strategy()` header overrides use a separate file: + +```json +{ + "default_qty_value": 5, + "commission_value": 0.04 +} +``` + +```bash +--strategy-overrides overrides.json +``` + +Input and override values sent to the FastAPI service must be scalar strings, +numbers, or booleans. + +## Runtime image policy + +The package default is an exact `pineforge-release` version and OCI digest. The +`missing` pull policy downloads it only when absent; `never` supports offline +runs; `always` refreshes a tag before running. + +The rolling channel is explicit: + +```bash +--runtime-image ghcr.io/pineforge-4pass/pineforge-release:latest \ +--pull-policy always +``` + +Use the pinned default for reproducible research. A rolling tag may change +engine or codegen behavior without a `pineforge-data` version change. + +## Report envelope + +The harness combines provider provenance with the release report: + +```json +{ + "schema_version": 1, + "request_id": null, + "provider": { + "name": "ccxt:kraken", + "adapter": "ccxt", + "venue": "kraken", + "source_timeframe": "15m", + "market": { + "symbol": "BTC/USD", + "provider_id": "XXBTZUSD", + "market_type": "spot", + "contract": null + } + }, + "data": { + "requested_start_ms": 1751328000000, + "requested_end_ms": 1751932800000, + "first_bar_ms": 1751328000000, + "last_bar_ms": 1751931900000, + "bars": 672 + }, + "runtime": { + "mode": "local-container", + "release_image": "ghcr.io/...@sha256:..." + }, + "backtest": { + "summary": {}, + "trades": [], + "metrics": {}, + "equity_curve": [], + "fingerprint": {} + } +} +``` + +Remote responses include a request ID. Server runtime provenance also includes +the generated C++ digest, compile-cache key/hit, and engine/codegen/release +versions. Local runs record the resolved image digest and OCI component labels +when Docker exposes them. + +## Reproducibility checklist + +Retain: + +- raw Pine source and strategy input/override files; +- provider adapter and venue; +- exact resolved symbol and provider market ID; +- requested interval and normalized OHLCV snapshot or checksum; +- release image reference and resolved digest; +- report fingerprint and request ID. + +The pinned release currently does not expose trace collection. `--trace` fails +explicitly rather than producing an incomplete report. diff --git a/docs/data-model.md b/docs/data-model.md new file mode 100644 index 0000000..8364c7a --- /dev/null +++ b/docs/data-model.md @@ -0,0 +1,139 @@ +# Normalized data model + +The public records are frozen, slotted dataclasses. Invalid values fail at +construction time so provider errors do not silently reach a backtest. + +## Instrument identity + +An `Instrument` separates portable market meaning from provider identity. + +| Field | Meaning | +|---|---| +| `symbol` | provider-normalized public symbol used in requests | +| `venue` | exchange, broker environment, or data-source instance | +| `provider_id` | raw provider/venue market ID retained for provenance | +| `asset_class` | crypto, equity, forex, commodity, index, fund, bond, or unknown | +| `market_type` | spot, cash, swap, future, option, CFD, or unknown | +| `base`, `quote`, `settle` | catalog-supplied assets/currencies | +| `contract` | derivative terms, or `None` for non-contract markets | +| `timezone` | IANA timezone used by Pine date/session behavior | +| `session` | Pine-compatible session description | +| `volume_unit` | meaning of bar/trade volume, such as base or contracts | + +Do not parse `symbol` to derive the other fields. Providers must populate them +from their market catalog. + +### Spot example + +```python +from pineforge_data import AssetClass, Instrument, MarketType + +spot = Instrument( + symbol="BTC/USD", + venue="kraken", + provider_id="XXBTZUSD", + asset_class=AssetClass.CRYPTO, + market_type=MarketType.SPOT, + base="BTC", + quote="USD", +) +``` + +### Contract example + +```python +from pineforge_data import ContractSpec, Instrument, MarketType + +swap = Instrument( + symbol="BTC/USDT:USDT", + venue="exchange", + market_type=MarketType.SWAP, + base="BTC", + quote="USDT", + settle="USDT", + volume_unit="contracts", + contract=ContractSpec( + contract_size=0.001, + linear=True, + inverse=False, + ), +) +``` + +`ContractSpec` can also carry `expiry_ms`, `strike`, and `option_type`. A +contract cannot be both linear and inverse, and numeric contract terms must be +finite and positive. + +## Market listings and discovery + +`MarketListing` wraps an instrument with venue capabilities: + +- `active`: whether the provider reports the listing as tradable; +- `margin_supported`: whether margin is available. + +Margin is a capability rather than a market type: a spot listing may support +margin while remaining `MarketType.SPOT`. + +`MarketQuery` filters listings by asset class, one or more market types, +base/quote/settlement asset, active state, margin support, and linear/inverse +contract form. Text asset filters are case-insensitive. + +## Bars + +`Bar` contains one normalized OHLCV candle: + +- `timestamp_ms` is a non-negative signed-64-bit Unix timestamp; +- OHLC prices are finite and positive; +- `high` must be at least every other OHLC price; +- `low` must be no greater than every other OHLC price; +- volume is finite and non-negative; +- `instrument` and `source` preserve provenance. + +Providers should emit bars in strictly increasing timestamp order. A +`BarRequest` defines an inclusive `start_ms`, exclusive `end_ms`, source +timeframe, and optional positive limit. + +## Live trades + +`TradeTick` carries timestamp, local sequence, price, quantity, source, and +instrument. Sequence values are unsigned-64-bit compatible and allow the engine +stream boundary to reject duplicates or out-of-order handoffs. + +`TradeSubscription.start_ms` selects the initial provider timestamp. Its +`start_sequence` is the last sequence already accepted downstream, so the next +emitted record uses `start_sequence + 1`. + +## Macro observations + +`MacroObservation` records three different times: + +| Timestamp | Meaning | +|---|---| +| `period_end_ms` | end of the measured economic period | +| `released_at_ms` | first time the value became public | +| `vintage_at_ms` | time this particular revision became available | + +The enforced ordering is `period_end_ms <= released_at_ms <= vintage_at_ms`. +Backtests must align on availability/vintage time rather than inserting today's +revised value into historical periods. + +`MacroDataProvider` and `MacroRequest` define the public contract; the bootstrap +package does not yet ship a built-in macro provider. + +## Low-level engine streaming + +Most raw-Pine users should use the local release container or FastAPI server. +For callers that already own a compiled strategy library and state handle, the +package also exposes dependency-free `ctypes` interoperability: + +- `pack_bars()` creates a contiguous `pf_bar_t` array; +- `pack_trade_ticks()` creates a contiguous `pf_trade_tick_t` array; +- `EngineStreamSink.begin()` warms and starts the engine stream; +- `push_tick()` and `push_ticks()` deliver normalized executions; +- `advance_time()` closes elapsed bars when no trade arrives; +- `end()` finishes the stream and can optionally finalize a partial input bar. + +All packed records must belong to the expected instrument. The caller owns the +native library and strategy state; `EngineStreamSink` does not create or free +them. Non-zero engine statuses raise `EngineStreamError` with the strategy's +last error message. diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..50a2642 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,107 @@ +# Getting started + +## Requirements + +- Python 3.11 or newer; +- Docker only when running raw-Pine backtests locally or building the FastAPI + server; +- network access to the selected data provider; +- provider credentials only when the requested endpoint requires them. + +No Git submodules, local C++ compiler, or engine checkout is required. + +## Install + +Install CCXT support for crypto exchanges: + +```bash +python3 -m venv .venv +.venv/bin/pip install 'pineforge-data[ccxt]' +``` + +Available extras: + +| Extra | Purpose | +|---|---| +| `ccxt` | CCXT async exchange adapter | +| `server` | FastAPI and Uvicorn server dependencies | +| `dev` | tests, type checking, formatting, and package builds | + +## Resolve a market and fetch confirmed bars + +Use the provider's normalized symbol exactly as returned by its market catalog. +For CCXT, spot `BTC/USDT` and linear swap `BTC/USDT:USDT` are different +instruments. + +```python +import asyncio + +from pineforge_data import BarRequest, CcxtProvider + + +async def main() -> None: + async with CcxtProvider("kraken") as provider: + listing = await provider.resolve_market("BTC/USD") + instrument = listing.instrument + bars = await provider.fetch_bars( + BarRequest( + instrument=instrument, + timeframe="15m", + start_ms=1_751_328_000_000, + end_ms=1_751_414_400_000, + ) + ) + print(instrument.market_type, instrument.provider_id, len(bars)) + + +asyncio.run(main()) +``` + +The CCXT adapter paginates, deduplicates timestamps, sorts the result, and +excludes a candle that was not closed at the request's observation time. + +## Run a raw-Pine backtest locally + +```bash +.venv/bin/pineforge-backtest \ + --pine strategy.pine \ + --provider ccxt \ + --venue kraken \ + --symbol BTC/USD \ + --timeframe 15m \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-08T00:00:00Z \ + --output report.json \ + --pretty +``` + +The first run pulls the package's digest-pinned `pineforge-release` image. The +provider runs on the host; only raw PineScript, normalized OHLCV, syminfo, and +runtime options enter the isolated container. + +## Use the FastAPI server + +If a backtest server is already running: + +```bash +export PINEFORGE_SERVER_URL=http://127.0.0.1:8000 +export PINEFORGE_SERVER_API_KEY=change-me +.venv/bin/pineforge-backtest \ + --pine strategy.pine \ + --venue kraken \ + --symbol BTC/USD \ + --timeframe 15m \ + --start 2025-07-01T00:00:00Z \ + --end 2025-07-08T00:00:00Z +``` + +The data fetch still happens locally. The harness sends normalized bars to the +server, which provides bounded concurrency and a generated-C++ keyed compile +cache. + +## Next steps + +- [Learn the normalized data model](data-model.md). +- [Search and filter provider markets](providers.md). +- [Configure parameters, runtime channels, and reports](backtesting.md). +- [Deploy the concurrent server](server.md). diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..e65fc43 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,55 @@ +# PineForge Data documentation + +PineForge Data connects external market and macro providers to deterministic +PineForge backtests. It owns provider transport, market discovery, normalized +records, data provenance, and the local or remote handoff to +`pineforge-release`. + +```text +exchange / broker / macro API + ↓ + Python provider adapter + ↓ + normalized market + bars + trades + ↓ + local pineforge-release or FastAPI server + ↓ + versioned report with data/runtime provenance +``` + +## Choose a guide + +| Goal | Guide | +|---|---| +| Install the package and run a first backtest | [Getting started](getting-started.md) | +| Understand instruments, contracts, bars, trades, and macro vintages | [Data model](data-model.md) | +| Discover markets and use CCXT or another provider | [Using providers](providers.md) | +| Configure local and remote raw-Pine backtests | [Backtesting](backtesting.md) | +| Deploy and operate the concurrent FastAPI service | [FastAPI server](server.md) | +| Implement a new exchange or broker adapter | [Provider contract](provider-contract.md) | +| Prepare a contribution | [Contributing](../CONTRIBUTING.md) | + +## Package boundaries + +- `pineforge-data` is Python-only and owns external data integration. +- `pineforge-release` owns Pine transpilation, C++ compilation, and the engine + runtime image. +- `pineforge-engine` receives normalized arrays and must not depend on provider + SDKs or vendor-specific schemas. +- Provider credentials stay on the data-fetching host. The FastAPI backtest + request contains normalized data, not provider credentials. + +## Core guarantees + +- timestamps use Unix milliseconds; +- normalized records retain their instrument and source provenance; +- historical bars exclude the currently forming candle when the provider can + identify it; +- exact catalog resolution distinguishes spot, swaps, futures, and options; +- macro observations retain release and vintage timestamps to avoid revised- + data lookahead; +- backtest reports identify both the resolved market and runtime versions. + +The package does not promise that every provider exposes every protocol. A +provider may support historical bars without live trades, or macro observations +without a market catalog. diff --git a/docs/providers.md b/docs/providers.md new file mode 100644 index 0000000..ae8dca5 --- /dev/null +++ b/docs/providers.md @@ -0,0 +1,138 @@ +# Using providers + +Providers are structural Python protocols rather than required base classes. +They are selected independently from venues: + +- provider: adapter implementation, such as `ccxt`; +- venue: an exchange or broker environment, such as `kraken`; +- symbol: the exact normalized market symbol on that provider. + +## Create a registered provider + +```python +from pineforge_data import create_provider + + +async def resolve_btc_usd(): + provider = create_provider("ccxt", "kraken", config={"enableRateLimit": True}) + try: + return await provider.resolve_market("BTC/USD") + finally: + await provider.close() +``` + +`ProviderRegistry` contains built-in factories and discovers third-party +packages through the `pineforge_data.providers` entry-point group. Unknown or +invalid adapters raise `ProviderNotFoundError` or `ProviderRegistryError`. + +## Discover CCXT markets + +```python +from pineforge_data import CcxtProvider, MarketQuery, MarketType + + +async def list_linear_swaps(): + async with CcxtProvider("okx") as provider: + swaps = await provider.list_markets( + MarketQuery( + market_types=frozenset({MarketType.SWAP}), + quote="USDT", + settle="USDT", + active=True, + linear=True, + ) + ) + for listing in swaps[:10]: + print( + listing.instrument.symbol, + listing.instrument.provider_id, + listing.instrument.contract, + ) +``` + +Use `resolve_market()` before fetching data. Resolution is exact: passing a raw +exchange ID where a unified symbol is expected raises `MarketNotFoundError`. +This prevents an ambiguous ticker from silently choosing spot instead of a +swap, future, or option. + +## Fetch historical bars + +```python +from pineforge_data import BarRequest, CcxtProvider + + +async def fetch_swap_bars(): + async with CcxtProvider("okx") as provider: + listing = await provider.resolve_market("BTC/USDT:USDT") + return await provider.fetch_bars( + BarRequest( + instrument=listing.instrument, + timeframe="1h", + start_ms=1_751_328_000_000, + end_ms=1_751_587_200_000, + limit=500, + ) + ) +``` + +CCXT behavior: + +- requires the exchange's unified `fetchOHLCV` capability; +- paginates by timestamp up to the requested limit; +- deduplicates repeated candle timestamps; +- sorts results in ascending order; +- excludes bars outside `[start_ms, end_ms)`; +- excludes a candle whose close time is later than the observation time. + +Unsupported methods raise `CcxtCapabilityError`; malformed records raise +`CcxtDataError`; missing optional CCXT installation raises +`CcxtDependencyError`. + +## Stream public trades + +```python +from pineforge_data import CcxtProvider, TradeSubscription + + +async def print_next_trade(): + async with CcxtProvider("kraken") as provider: + listing = await provider.resolve_market("BTC/USD") + subscription = TradeSubscription( + instrument=listing.instrument, + start_ms=1_751_328_000_000, + start_sequence=42, + ) + async for tick in provider.stream_trades(subscription): + print(tick.sequence, tick.timestamp_ms, tick.price, tick.quantity) + break +``` + +The bootstrap CCXT adapter polls the unified public-trades endpoint, orders each +batch by timestamp, deduplicates by provider trade ID (or normalized values when +no ID exists), and assigns a strictly increasing local sequence. Stop the async +iterator when handing control back to the caller and close the provider. + +## Provider configuration + +`CcxtProvider` accepts separate configuration layers: + +| Argument | Purpose | +|---|---| +| `config` | CCXT exchange constructor options and credentials | +| `market_params` | exchange-specific `load_markets` parameters | +| `ohlcv_params` | exchange-specific OHLCV endpoint parameters | +| `trade_params` | exchange-specific public-trade parameters | +| `page_limit` | maximum records requested per REST page | +| `poll_interval_ms` | delay between public-trade polls | +| `dedup_window` | retained trade identity window | + +Keep credentials outside source control. The CLI accepts constructor options +through `--provider-config`; programmatic callers can use all endpoint-specific +arguments. + +## Implement another provider + +Read the [provider contract](provider-contract.md). Backtest-compatible +providers implement `MarketDataProvider`: market listing, exact resolution, +historical bars, and cleanup. Live trades and macro data remain separate, +optional protocols. diff --git a/tests/test_documentation.py b/tests/test_documentation.py new file mode 100644 index 0000000..eb7bba6 --- /dev/null +++ b/tests/test_documentation.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +import re +from pathlib import Path +from urllib.parse import urlsplit + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +MARKDOWN_LINK = re.compile(r"\[[^]]*]\(([^)]+)\)") +DOCUMENTS = [ROOT / "README.md", ROOT / "CONTRIBUTING.md", *sorted((ROOT / "docs").glob("*.md"))] + + +@pytest.mark.parametrize("document", DOCUMENTS, ids=lambda path: path.name) +def test_relative_documentation_links_exist(document: Path) -> None: + for raw_target in MARKDOWN_LINK.findall(document.read_text(encoding="utf-8")): + target = raw_target.split("#", 1)[0] + if not target or urlsplit(target).scheme: + continue + resolved = (document.parent / target).resolve() + assert resolved.exists(), f"broken link in {document.relative_to(ROOT)}: {raw_target}"