Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ extraction targets and should not be imported until they exist.
| `ordersim/connectors/csv.py` | Normalized CSV `MBOEvent` source | Yes |
| `ordersim/connectors/databento.py` | Databento MBO normalization | Yes |
| `ordersim/connectors/parquet.py` | Normalized Parquet `MBOEvent` source | Yes |
| `ordersim/connectors/binance/` | Binance L2 capture and integrity boundary | Capture API only |
| `ordersim/connectors/binance/` | Binance capture and typed L2 source; not MBO | Public venue API |
| `ordersim/latency.py` | Latency model contracts and reference models | Yes |
| `ordersim/replay/simulator.py` | Replay orchestration and `run_many` | Yes |
| `ordersim/testing/` | Public helpers for extension tests | Public |
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ All notable public changes to `ordersim` are documented here.
aggregate-trade, book-ticker, snapshot, and RPI evidence.
- Added connection manifests and explicit diff-depth sequence-gap records,
while keeping lower-fidelity capture separate from modeled MBO replay.
- Added a typed, streaming reader for completed Binance captures with exact
depth, aggregate-trade, and book-ticker records.
- Added snapshot bridging and `pu`/`u` continuity validation for standard
Binance diff-depth segments.

## 0.1.3 - 2026-05-20

Expand Down
7 changes: 5 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,19 +69,22 @@ Lower-fidelity venue data takes a longer, explicit path:
flowchart LR
venue["Venue L2 + trades"]
capture["Raw capture"]
source["Typed L2 source"]
model["Named reconstruction model"]
modeled["Modeled MBOEvent stream"]
parquet["Canonical Parquet + model manifest"]
replay["Replay"]

venue --> capture --> model --> modeled --> parquet --> replay
venue --> capture --> source --> model --> modeled --> parquet --> replay
```

Capture code may live beside connectors because it owns venue I/O and source
schemas. Capture alone is not a `DataSource`: observed L2 rows must not be
presented as exchange-native MBO. The reconstruction model owns that
lower-fidelity assumption and must preserve a manifest describing how its
events were inferred.
events were inferred. For Binance, `BinanceCaptureSource` is the typed,
sequence-validated L2 boundary between the raw evidence and that future model;
it does not implement the canonical MBO `DataSource` protocol.

## One Replay Run

Expand Down
55 changes: 52 additions & 3 deletions docs/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,13 +111,62 @@ connection segment and obtains a new REST snapshot.

These files are intentionally not canonical replay data. Binance depth has no
stable public order IDs, and individual additions and cancellations inside an
update window are not observable. A future named L2-to-virtual-L3 model will
consume the capture, document the inference policy, and only then emit modeled
`MBOEvent` rows.
update window are not observable. A named L2-to-virtual-L3 model must consume
the typed capture records, document the inference policy, and only then emit
modeled `MBOEvent` rows.

Capture files are local research data and must not be committed to the
repository.

### Reading Completed Captures

`BinanceCaptureSource` streams completed gzip capture files directly, so a
multi-day capture does not need to be loaded into memory:

```python
from ordersim.connectors.binance import BinanceCaptureSource

source = BinanceCaptureSource.from_manifest(
"captures/binance/manifest-20260728T105500Z.json"
)

for event in source.validated_depth_events():
print(event)

for trade in source.aggregate_trades():
print(trade)
```

This is a typed Binance source, not the canonical `DataSource` protocol. It
emits `BinanceDepthSnapshot`, `BinanceDepthUpdate`,
`BinanceAggregateTrade`, and `BinanceBookTicker` records rather than
`MBOEvent`. Passing it directly to `Replay` is intentionally unsupported.

The reader preserves prices and quantities as exact `Decimal` values. Binance
exchange event and transaction timestamps (`E` and `T`) are milliseconds since
the Unix epoch and are normalized to UTC nanoseconds. Local UTC receive and
monotonic receive timestamps from the capture envelope remain available
separately.

`validated_depth_events()` applies Binance's USD-M synchronization rules per
connection:

1. Require a REST snapshot before standard diff-depth updates.
2. Discard buffered updates where `u < lastUpdateId`.
3. Require the first retained update to satisfy
`U <= lastUpdateId <= u`.
4. Require each later update's `pu` to equal the preceding update's `u`.

A broken segment raises `BinanceSequenceError`; it is never repaired silently.
Depth quantities are absolute, and a zero quantity means remove that price
level. RPI depth is available separately through
`depth_updates(stream_kind="rpi_depth")`; it is not merged into standard depth
by the source.

Aggregate trades preserve Binance's optional `nq` field as
`normal_quantity`. When present, it is the quantity excluding trades involving
RPI orders. When absent, `normal_quantity` is `None`, not an inferred value.

For the user-facing decision guide, see `docs/data-guide.md`.

## In-Memory Sources
Expand Down
16 changes: 14 additions & 2 deletions docs/data-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,20 @@ the connector and decide whether the connector is valid for the research task.

Binance USD-M depth is one such lower-fidelity source. The Binance capture tool
records raw L2 depth, aggregate trades, and integrity metadata, but its output
is not accepted by `Replay` as observed MBO. See `docs/connectors.md` for the
capture boundary and the planned modeled reconstruction path.
is not accepted by `Replay` as observed MBO.

After a capture completes, use `BinanceCaptureSource` to stream exact typed
snapshots, sequence-validated depth updates, aggregate trades, and book
tickers. That typed source is the input boundary for the planned named
virtual-L3 reconstruction model:

```text
raw capture -> BinanceCaptureSource -> named model -> modeled MBO + manifest
```

There is deliberately no direct
`BinanceCaptureSource -> Replay` path. See `docs/connectors.md` for the capture
and validation contract.

## Related Docs

Expand Down
28 changes: 28 additions & 0 deletions docs/schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,34 @@ before replay sees the event.
The replay layer operates only on normalized integer timestamps. It does not
carry Python timezone objects or infer timezone rules after normalization.

## Binance L2 Records

The Binance connector exposes typed records before the modeled-MBO boundary.
These records describe observed aggregated depth and trades; they do not claim
to contain stable exchange order IDs.

`BinancePriceLevel` holds an exact positive `Decimal` price and a non-negative
`Decimal` quantity. In a depth update, the quantity is the new absolute
quantity at that price; zero means remove the level.

| Record | Important fields | Meaning |
|---|---|---|
| `BinanceDepthSnapshot` | `last_update_id`, `bids`, `asks` | REST depth state anchoring one connection. |
| `BinanceDepthUpdate` | `first_update_id`, `final_update_id`, `previous_update_id`, `bids`, `asks` | One standard or RPI absolute-quantity diff-depth message. |
| `BinanceAggregateTrade` | `aggregate_trade_id`, `price`, `quantity`, `normal_quantity`, `buyer_is_maker` | Trades aggregated by price and taking side. |
| `BinanceBookTicker` | `update_id`, bid and ask price/quantity | Real-time best bid and ask observation. |

All records include `symbol`, `connection_id`, UTC receive nanoseconds, and
local monotonic receive nanoseconds. Stream messages also include exchange
event and transaction/trade times normalized from Binance milliseconds to UTC
nanoseconds.

Binance contract quantity can be fractional, so the connector preserves it as
`Decimal`. A future virtual-L3 reconstruction model must declare its quantity
unit and exact conversion rule before producing the canonical integer
`MBOEvent.size`. These L2 records are therefore not accepted directly by
`Replay`.

## `MBOEvent`

`MBOEvent` represents one Level 3 / market-by-order event.
Expand Down
24 changes: 24 additions & 0 deletions src/ordersim/connectors/binance/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,34 @@
"""

from ordersim.connectors.binance.capture import capture_binance
from ordersim.connectors.binance.l2 import (
BinanceAggregateTrade,
BinanceBookTicker,
BinanceCaptureEnvelope,
BinanceDepthEvent,
BinanceDepthSnapshot,
BinanceDepthUpdate,
BinancePriceLevel,
DepthStreamKind,
)
from ordersim.connectors.binance.schema import BinanceCaptureConfig, CaptureManifest
from ordersim.connectors.binance.source import (
BinanceCaptureSource,
BinanceSequenceError,
)

__all__ = [
"BinanceAggregateTrade",
"BinanceBookTicker",
"BinanceCaptureConfig",
"BinanceCaptureEnvelope",
"BinanceCaptureSource",
"BinanceDepthEvent",
"BinanceDepthSnapshot",
"BinanceDepthUpdate",
"BinancePriceLevel",
"BinanceSequenceError",
"CaptureManifest",
"DepthStreamKind",
"capture_binance",
]
Loading