diff --git a/AGENTS.md b/AGENTS.md index 446487a..0ead098 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,6 +92,10 @@ gateway or whether the strategy should own that state itself. ### Add a Data Connector +Keep the core replay boundary venue-, asset-class-, and vendor-independent. +Databento and Binance are current reference integrations, not architectural +special cases. + 1. Implement the public `DataSource` protocol. 2. Convert source data into the canonical MBO event schema. 3. Add a tiny fixture or generator that does not require private data. @@ -157,8 +161,8 @@ Python engine before release: - same order-intent log where the public API observes it. Engines consume normalized `MBOEvent` rows. Do not make an engine responsible -for reading Databento, CSV, Parquet, or any other source format; that belongs in -a connector. +for reading vendor data, CSV, Parquet, or any other source format; that belongs +at the connector or reconstruction boundary. Use `ordersim.testing.assert_equivalent_execution_engines` before trusting a new engine implementation. diff --git a/CHANGELOG.md b/CHANGELOG.md index ef2b645..e10f087 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,9 @@ All notable public changes to `ordersim` are documented here. trades aligned, every processed L2 endpoint reconciled, every joinable book-ticker state matched, and an open challenge to compare fill models against paired L3 or live execution evidence. +- Clarified the vendor- and asset-class-independent architecture: observed MBO + and reconstructed virtual MBO are two explicit fidelity paths, with + Databento and Binance documented as their current reference integrations. ## 0.1.3 - 2026-05-20 diff --git a/README.md b/README.md index 7dbec36..8f5232f 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,32 @@ interest include order-book replay, market replay, latency modeling, queue-position effects, fill simulation, execution modeling, execution-aware backtesting, and market microstructure research. -## Crypto L2, Reconstructed for MBO Replay +The replay core is independent of asset class, venue, and data vendor. It +operates on one canonical `MBOEvent` contract; integrations are responsible for +reaching that boundary without hiding the fidelity of their source data. + +## Two Data Fidelity Paths + +`ordersim` supports two explicit routes into the same queue-aware replay: + +| Path | Source evidence | Current reference implementation | Replay input | +|---|---|---|---| +| Observed MBO | Order-level events with stable order IDs | Databento MBO | Observed `MBOEvent` rows | +| Reconstructed virtual MBO | Sequence-valid L2 depth plus individual trades | Binance USD-M | Modeled `MBOEvent` rows plus a reconstruction manifest | + +Databento and Binance are the integrations available today, not boundaries of +the architecture. Futures, crypto, and equity datasets from additional venues +and vendors should join through one of these fidelity paths without changing +the replay or execution-engine APIs. Normalized CSV and Parquet remain +vendor-neutral interchange and storage formats. + +### Observed MBO + +When a source provides stable order IDs, its connector can normalize the +observed add, cancel, modify, and trade events directly into `MBOEvent`. This is +the highest-fidelity path. Databento is the current reference implementation. + +### Reconstructed Virtual MBO for Crypto L2 Most crypto venues publish market-by-price depth, not the stable order IDs needed for true market-by-order replay. `ordersim` takes a different path: it @@ -28,7 +53,7 @@ price-level endpoint, and emits a deterministic **virtual MBO** stream under explicit queue assumptions. That means crypto data can use the same inspectable, queue-aware Python/C++ -execution engines as native MBO data without pretending the inferred orders +execution engines as observed MBO data without pretending the inferred orders were observed at the exchange. The first full Binance USD-M study reconstructed: @@ -43,7 +68,7 @@ and book consistency, not knowledge of Binance's hidden FIFO queue. The recommended conservative policy and an optimistic sensitivity policy expose that uncertainty instead of burying it inside one fill rule. -### The Crypto Realism Challenge +#### The Crypto Realism Challenge Have a more realistic L2 execution model? Compare it with evidence. Use paired L2/L3 data or live passive-order outcomes, disclose the queue and latency @@ -65,8 +90,8 @@ join the [public validation challenge](https://github.com/tradingexpert/ordersim strategy's orders, position, and portfolio state isolated. - Exposes a small, regular Python API that is easy to read, debug, test, and extend. -- Captures Binance L2 and individual-trade evidence and reconstructs modeled - MBO under explicit conservative and optimistic queue assumptions. +- Accepts observed MBO directly or reconstructed virtual MBO produced from + lower-fidelity evidence under an explicit, named model. ## What It Is Not @@ -161,7 +186,8 @@ Install the current release from PyPI with: pip install ordersim ``` -Optional data integrations and file formats are installed separately as extras: +Current optional integrations and file formats are installed separately as +extras: ```bash pip install "ordersim[databento]" @@ -177,8 +203,9 @@ from ordersim import CsvSource source = CsvSource("events.csv") ``` -For repeated research runs, the recommended path is to normalize once, -materialize the canonical Parquet form, and replay from that thereafter: +For repeated research runs, the recommended path is to reach the canonical +`MBOEvent` boundary once, materialize Parquet, and replay from that thereafter. +For example, the current observed-MBO integration uses Databento: ```python import databento as db @@ -192,8 +219,8 @@ write_parquet(raw_source, "events.parquet") source = ParquetSource("events.parquet") ``` -Direct raw-source replay is still useful for one-off inspection and connector -development: +Direct source replay is still useful for one-off inspection and connector +development. With the current Databento integration: ```python import databento as db @@ -302,9 +329,11 @@ same input. ## Status `0.1.x` is live on PyPI. The current public line includes the Python reference -engine, packaged C++ default, canonical connector -> Parquet -> replay workflow, +engine, packaged C++ default, canonical `MBOEvent` -> Parquet -> replay workflow, latency models, economics, public execution-equivalence fixtures, and an -evidence-first Binance L2-to-virtual-MBO research path. +observed-MBO connector for Databento plus an evidence-first Binance USD-M +L2-to-virtual-MBO reference implementation. Both integrations feed the same +vendor- and asset-class-independent replay boundary. Planned next milestones: diff --git a/docs/architecture.md b/docs/architecture.md index 77eac96..d16b234 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -8,8 +8,8 @@ the rest. ```mermaid flowchart LR - raw["Raw vendor data"] - connector["Connector
DatabentoMboSource / CsvSource / ParquetSource"] + evidence["Source evidence
observed MBO or L2 + trades"] + integration["Connector or
named reconstruction model"] canonical["Canonical MBOEvent stream"] replay["Replay"] recording["RecordingGateway"] @@ -21,8 +21,8 @@ flowchart LR valuation["Valuation marks"] result["ReplayResult
fills, order log, economics"] - raw --> connector - connector --> canonical + evidence --> integration + integration --> canonical canonical --> replay replay --> recording recording <--> strategy @@ -38,19 +38,26 @@ flowchart LR The main boundaries are: -- connectors normalize external data into `MBOEvent`; +- observed-MBO connectors normalize order-level data directly into `MBOEvent`; +- lower-fidelity sources require a named reconstruction model before they + reach the canonical boundary; - strategies depend on `OrderGateway`, not on storage or engine internals; - replay normalizes inputs once, chooses an execution engine, and gathers results; - valuation marks are collected during replay and consumed by economics; - the Python engine defines behavior; the C++ engine must match it. -## Recommended Data Flow +## Data Fidelity Paths + +The replay core is independent of venue, asset class, and data vendor. Two data +paths converge on the canonical `MBOEvent` boundary. + +Observed order-level data takes the direct path: ```mermaid flowchart LR - raw["Raw vendor source"] - normalize["Vendor connector"] + raw["Observed order-level source"] + normalize["MBO connector"] materialize["write_parquet(...)"] parquet["Canonical Parquet"] source["ParquetSource"] @@ -61,16 +68,17 @@ flowchart LR Direct connector replay is supported, but repeated research should normally materialize canonical Parquet once and replay from it thereafter. See -`docs/data-guide.md`. +`docs/data-guide.md`. Databento is the current reference connector for this +path; CSV and Parquet support the normalized canonical schema. -Lower-fidelity venue data takes a longer, explicit path: +Price-level data takes a longer, explicit path: ```mermaid flowchart LR - venue["Venue L2 + aggregate and individual trades"] + venue["Venue L2 + individual trades"] capture["Raw capture"] source["Typed L2 source"] - model["BinanceMBOReconstructor
named queue policy"] + model["Named reconstruction model
explicit queue policy"] modeled["Modeled MBOEvent stream"] parquet["Canonical Parquet + model manifest"] replay["Replay"] @@ -78,9 +86,10 @@ flowchart LR venue --> capture --> source --> model --> modeled --> parquet --> replay ``` -This is the crypto-enabling path: aggregated venue data reaches the same -canonical event and execution-engine boundary as native MBO, but only after a -named model makes the missing order-level structure explicit. New +This is the path that currently enables Binance crypto data: aggregated venue +data reaches the same canonical event and execution-engine boundary as observed +MBO, but only after a named model makes the missing order-level structure +explicit. The architecture itself is not Binance- or crypto-specific. New reconstruction policies should compete on paired-L3 or live-fill prediction, not on undocumented claims of realism. @@ -88,10 +97,11 @@ Capture code may live beside connectors because it owns venue I/O and source schemas. Capture alone is not a `DataSource`: observed L2 rows must not be presented as exchange-native MBO. The reconstruction model owns that lower-fidelity assumption and preserves a study manifest describing how its -events were inferred. For Binance, `BinanceCaptureSource` is the typed, -sequence-validated L2 boundary and `BinanceMBOReconstructor` emits modeled -canonical events for one snapshot-anchored segment. The capture source itself -does not implement the canonical MBO `DataSource` protocol. +events were inferred. The current reference implementation uses +`BinanceCaptureSource` as the typed, sequence-validated L2 boundary and +`BinanceMBOReconstructor` to emit modeled canonical events for one +snapshot-anchored segment. The capture source itself does not implement the +canonical MBO `DataSource` protocol. ## One Replay Run diff --git a/docs/assumptions.md b/docs/assumptions.md index 60d7257..166295a 100644 --- a/docs/assumptions.md +++ b/docs/assumptions.md @@ -7,7 +7,14 @@ replay under explicitly named assumptions. The goal is not to claim perfect market realism. The goal is to make every assumption visible, testable, and replaceable. -## Data Assumptions +## Data Fidelity Assumptions + +The replay and execution engines do not distinguish asset classes or vendors. +They distinguish the evidence behind their canonical events: order-level +events may be observed directly, or they may be reconstructed from +lower-fidelity evidence under a named model. + +### Observed MBO The highest-fidelity path expects order-level data: @@ -26,6 +33,8 @@ Public timestamps are normalized as UTC Unix-epoch nanoseconds. Connectors own timezone-aware conversion from vendor or exchange-local source timestamps before replay sees the data. +### Reconstructed Virtual MBO + Lower-fidelity data can be supported, but must be named honestly. A Level 2 or MBP source provides aggregated size by price level, not individual queue members. Queue-position behavior inferred from MBP is necessarily a model, not @@ -43,8 +52,8 @@ must preserve the observed price-level updates and trades. Reconstruction must name its cancellation-allocation, event-ordering, and randomness assumptions, and modeled events must not be presented as exchange-native MBO. -The Binance virtual-MBO model uses the minimum-flow identity at each price -level and depth interval: +The current Binance reference model uses the minimum-flow identity at each +price level and depth interval: ```text ending quantity = starting quantity + adds - cancels - traded quantity diff --git a/docs/connectors.md b/docs/connectors.md index 3f53c51..4a9d8c0 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -1,6 +1,14 @@ # Connectors -Connectors translate vendor data into the public `MBOEvent` schema. +Data integrations enter `ordersim` through one of two fidelity boundaries: + +- observed-MBO connectors normalize order-level source data directly into the + public `MBOEvent` schema; +- evidence connectors preserve lower-fidelity source records for a named + reconstruction model, which then emits modeled `MBOEvent` rows. + +The distinction is deliberate. A source integration must not make L2 data look +like observed MBO simply to satisfy the replay interface. The public contract is intentionally small: @@ -9,8 +17,21 @@ class DataSource(Protocol): def events(self) -> Iterable[MBOEvent]: ... ``` -A connector should hide SDK details, file layout, network access, and vendor -column names. Replay code should receive only normalized `MBOEvent` rows. +At the canonical boundary, a connector should hide SDK details, file layout, +network access, and vendor column names. Replay code should receive only +normalized `MBOEvent` rows. + +## Current Integrations + +| Integration | Fidelity role | Output | +|---|---|---| +| `DatabentoMboSource` | Observed order-level reference | Canonical `MBOEvent` | +| `CsvSource` / `ParquetSource` | Vendor-neutral normalized interchange | Canonical `MBOEvent` | +| `BinanceCaptureSource` | L2 and trade evidence reference | Typed evidence for reconstruction | +| `BinanceMBOReconstructor` | Named virtual-MBO model | Modeled `MBOEvent` plus manifest | + +These are the implementations available today, not permanent asset-class, +venue, or vendor boundaries. Normalized `ts_ns` values must be UTC Unix-epoch nanoseconds. Connectors may read exchange-local, vendor-local, or already-UTC source timestamps, but they @@ -37,7 +58,7 @@ Every connector should include a tiny public fixture or generator and at least one deterministic replay test. The test should prove the connector can produce events that a strategy can replay without private data. -## Recommended Workflow +## Canonical Storage Workflow For repeated research, normalize raw vendor data once, materialize the canonical Parquet form, and replay from `ParquetSource` thereafter: @@ -63,7 +84,7 @@ Direct connector replay remains useful for smoke tests, one-off inspection, and connector development. CSV remains useful for tiny examples and reviewable fixtures. -## Binance L2 Capture +## Reconstructed-MBO Reference: Binance USD-M Binance USD-M futures publishes aggregated price-level depth rather than stable market-by-order identifiers. The capture tool therefore records source evidence @@ -360,7 +381,7 @@ from ordersim import write_parquet write_parquet(source, "events.parquet") ``` -## Databento MBO Sources +## Observed-MBO Reference: Databento Use `DatabentoMboSource` with raw Databento MBO records, such as records yielded by an iterable `DBNStore`: diff --git a/docs/data-guide.md b/docs/data-guide.md index 5b7ae55..19d2ba9 100644 --- a/docs/data-guide.md +++ b/docs/data-guide.md @@ -1,15 +1,25 @@ # Data Guide -The default `ordersim` data path is: +`ordersim` has two source-fidelity paths that converge on the same canonical +replay boundary: ```text -raw vendor data -> connector -> canonical Parquet -> repeated replay +observed MBO -> connector -----------------------> MBOEvent +L2 + individual trades -> named reconstruction -> MBOEvent + model manifest + | + v + canonical Parquet -> repeated replay ``` -Normalize once at the boundary. Persist the normalized result. Replay the +The first path preserves observed order-level events. The second constructs an +explicit virtual order-level history from lower-fidelity evidence. Databento +and Binance are the current reference integrations for those paths; neither is +part of the replay core's identity. + +Reach the canonical boundary once. Persist the normalized result. Replay the canonical result. -That path keeps vendor-specific choices out of strategy experiments and gives +These paths keep vendor-specific choices out of strategy experiments and give repeated runs one durable local format. ## Which Source To Use @@ -17,7 +27,8 @@ repeated runs one durable local format. | Situation | Use | |---|---| | Repeated research over a real dataset | `ParquetSource` | -| First normalization from a vendor format | vendor connector, then `write_parquet(...)` | +| Source supplies stable order-level events | MBO connector, then `write_parquet(...)` | +| Source supplies L2 plus individual trades | typed source plus named reconstruction model | | One-off connector smoke test or inspection | direct connector replay | | Tiny human-readable example | `CsvSource` | | Test or package fixture | `InMemorySource` | @@ -25,9 +36,9 @@ repeated runs one durable local format. `CsvSource` is deliberately simple and reviewable. It is not the preferred storage format for large research datasets. -## Recommended Workflow +## Observed-MBO Example: Databento -For a vendor source such as Databento: +Databento is the current reference implementation of the observed-MBO path: ```python import databento as db @@ -102,7 +113,10 @@ All canonical sources must preserve the public `MBOEvent` contract: If a vendor source cannot preserve one of those properties, document the loss in the connector and decide whether the connector is valid for the research task. -Binance USD-M depth is one such lower-fidelity source. The Binance capture tool +## Reconstructed-MBO Example: Binance USD-M + +Binance USD-M depth is the current reference implementation for a +lower-fidelity source. The Binance capture tool records raw L2 depth, individual and aggregate trades, and integrity metadata, but its output is not accepted by `Replay` as observed MBO. diff --git a/docs/execution-engines.md b/docs/execution-engines.md index 0f6c1e0..fe0b8bb 100644 --- a/docs/execution-engines.md +++ b/docs/execution-engines.md @@ -15,13 +15,17 @@ They do not read vendor data directly. The architecture is: ```text -vendor data -> DataSource -> MBOEvent -> Replay -> ExecutionEngine +observed MBO -> connector -----------------------> MBOEvent +L2 + trades -> named reconstruction model ------> MBOEvent + | + v + Replay -> ExecutionEngine ``` -That means a Databento connector and a C++ execution engine are different +That means a source integration and a C++ execution engine are different extension points: -- a connector normalizes source data into `MBOEvent`; +- a connector or reconstruction model reaches the `MBOEvent` boundary; - an execution engine decides how strategy orders interact with those events. ## Reference Engine diff --git a/docs/schema.md b/docs/schema.md index 298e9bd..ab01115 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -17,7 +17,19 @@ before replay sees the event. The replay layer operates only on normalized integer timestamps. It does not carry Python timezone objects or infer timezone rules after normalization. -## Binance L2 Records +## Source Fidelity Boundaries + +Canonical sources emit `MBOEvent` rows. They may derive from observed +order-level data or from a named reconstruction model, but the provenance must +remain explicit. Typed L2 records are pre-canonical evidence: they cannot enter +`Replay` until a reconstruction policy has supplied the missing order-level +structure. + +The schema is vendor-independent at the replay boundary. The sections below +document the current Binance evidence records and the canonical event contract; +future integrations should preserve the same distinction. + +## Binance L2 Evidence Records The Binance connector exposes typed records before the modeled-MBO boundary. These records describe observed aggregated depth and trades; they do not claim