Add Binance L2 virtual-MBO reconstruction study

AGENTS.md

@@ -40,6 +40,8 @@ extraction targets and should not be imported until they exist.
 | `ordersim/connectors/parquet.py` | Normalized Parquet `MBOEvent` source | Yes |
 | `ordersim/connectors/binance/` | Binance capture and typed L2 source; not MBO | Public venue API |
 | `ordersim/connectors/binance/raw_trades.py` | Individual trade capture with ID-gap evidence | Public capture API |
+| `ordersim/connectors/binance/reconstruction.py` | Named virtual-MBO reconstruction policies | Public model API |
+| `ordersim/connectors/binance/reconstruction_study.py` | Streaming alignment and evidence report | Public research CLI |
 | `ordersim/latency.py` | Latency model contracts and reference models | Yes |
 | `ordersim/replay/simulator.py` | Replay orchestration and `run_many` | Yes |
 | `ordersim/testing/` | Public helpers for extension tests | Public |

CHANGELOG.md

@@ -20,6 +20,12 @@ All notable public changes to `ordersim` are documented here.
   reconciliation evidence.
 - Added typed `BinanceRawTrade` records alongside aggregate trades so the more
   detailed public evidence is available to future reconstruction models.
+- Added deterministic Binance L2-to-virtual-MBO reconstruction with explicit
+  queue-conservative and queue-optimistic policies, exact quantity scaling,
+  and canonical `MBOEvent` output.
+- Added a streaming reconstruction study that aligns individual trades to
+  depth intervals, validates joinable book-ticker states, preserves reconnect
+  boundaries, and reports inferred flow and model sensitivity.
 
 ## 0.1.3 - 2026-05-20
 

README.md

@@ -30,6 +30,8 @@ backtesting, and market microstructure research.
   strategy's orders, position, and portfolio state isolated.
 - Exposes a small, regular Python API that is easy to read, debug, test, and
   extend.
+- Captures Binance L2 and individual-trade evidence and reconstructs modeled
+  MBO under explicit conservative and optimistic queue assumptions.
 
 ## What It Is Not
 
@@ -129,6 +131,7 @@ Optional data integrations and file formats are installed separately as extras:
 ```bash
 pip install "ordersim[databento]"
 pip install "ordersim[parquet]"
+pip install "ordersim[binance]"
 ```
 
 Normalized CSV input works without optional dependencies:
@@ -265,7 +268,8 @@ same input.
 
 `0.1.x` is live on PyPI. The current public line includes the Python reference
 engine, packaged C++ default, canonical connector -> Parquet -> replay workflow,
-latency models, economics, and public execution-equivalence fixtures.
+latency models, economics, public execution-equivalence fixtures, and an
+evidence-first Binance L2-to-virtual-MBO research path.
 
 Planned next milestones:
 
@@ -284,6 +288,7 @@ Planned next milestones:
 - Architecture: `docs/architecture.md`
 - Data guide: `docs/data-guide.md`
 - Connectors: `docs/connectors.md`
+- Binance reconstruction study: `docs/binance-reconstruction-study.md`
 - Releasing: `docs/releasing.md`
 - Engineering standards: `docs/engineering-standards.md`
 - Benchmarks: `docs/benchmarks.md`

docs/architecture.md

@@ -70,7 +70,7 @@ flowchart LR
     venue["Venue L2 + aggregate and individual trades"]
     capture["Raw capture"]
     source["Typed L2 source"]
-    model["Named reconstruction model"]
+    model["BinanceMBOReconstructor<br/>named queue policy"]
     modeled["Modeled MBOEvent stream"]
     parquet["Canonical Parquet + model manifest"]
     replay["Replay"]
@@ -81,10 +81,11 @@ flowchart LR
 Capture code may live beside connectors because it owns venue I/O and source
 schemas. Capture alone is not a `DataSource`: observed L2 rows must not be
 presented as exchange-native MBO. The reconstruction model owns that
-lower-fidelity assumption and must preserve a manifest describing how its
+lower-fidelity assumption and preserves a study manifest describing how its
 events were inferred. For Binance, `BinanceCaptureSource` is the typed,
-sequence-validated L2 boundary between the raw evidence and that future model;
-it does not implement the canonical MBO `DataSource` protocol.
+sequence-validated L2 boundary and `BinanceMBOReconstructor` emits modeled
+canonical events for one snapshot-anchored segment. The capture source itself
+does not implement the canonical MBO `DataSource` protocol.
 
 ## One Replay Run
 

docs/assumptions.md

@@ -36,6 +36,25 @@ must preserve the observed price-level updates and trades. Reconstruction must
 name its cancellation-allocation, event-ordering, and randomness assumptions,
 and modeled events must not be presented as exchange-native MBO.
 
+The Binance virtual-MBO model uses the minimum-flow identity at each price
+level and depth interval:
+
+```text
+ending quantity = starting quantity + adds - cancels - traded quantity
+```
+
+It infers the smallest non-negative add and cancel quantities that satisfy that
+identity. This determines aggregate flow, but it does not determine where a
+cancel occurred inside the queue or exactly when inferred additions appeared
+inside Binance's update window.
+
+The recommended default is `queue-conservative`: additions needed by an
+interval are placed before its first trade and cancellations remove the newest
+modeled liquidity. This avoids selecting the assumptions most favorable to a
+passive strategy. `queue-optimistic` adds only when required and removes the
+oldest liquidity. Run both when queue position materially affects the result;
+their difference is model sensitivity, not measurement error.
+
 ## Queue Assumptions
 
 When the strategy places a resting limit order, the default model assumes the

docs/binance-reconstruction-study.md

@@ -0,0 +1,120 @@
+# Binance L2 Reconstruction Study
+
+This note records the first empirical validation of `ordersim`'s Binance
+L2-to-virtual-MBO path. It is evidence for a named model, not a claim that
+aggregated depth reveals Binance's true FIFO order queue.
+
+## Capture
+
+The study used locally captured Binance USD-M futures evidence from
+2026-08-08 through 2026-08-10:
+
+- standard full diff depth at 100 ms;
+- individual `@trade` messages;
+- real-time book ticker;
+- REST snapshots anchoring every depth connection;
+- aggregate trades, RPI depth, and REST individual trades retained as
+  supplemental audit evidence.
+
+The primary continuous window ends at `2026-08-10T17:12:57Z`, immediately
+before a machine-wide network outage. Later recovery records remain in the raw
+archive but were excluded. Raw capture files are research data and are not
+committed to this repository.
+
+## Method
+
+Individual trades are aligned to consecutive depth endpoints using Binance
+transaction timestamps. A 60-second local receive-time buffer allows the
+independent depth and trade connections to arrive out of order. This is an
+offline evidence-alignment window, not simulated exchange latency.
+
+The first full ETH pass showed why this must be measured: a two-second buffer
+left 13,422 trades late, and a ten-second buffer left 71. The 60-second pass
+left none; its maximum executable-trade receive delay was 29.121 seconds. Each
+study report records both maximum receive delay and any residual late-trade
+lag so the buffer can be reassessed for a different capture environment.
+
+For each side and price level:
+
+```text
+ending quantity = starting quantity + adds - cancels - traded quantity
+```
+
+The model infers the smallest non-negative add and cancel quantities satisfying
+that identity. It then applies two queue assumptions:
+
+- `queue-conservative`: infer additions before the first trade at the level and
+  cancel newest modeled liquidity first;
+- `queue-optimistic`: add only as needed and cancel oldest modeled liquidity
+  first.
+
+Both policies must reproduce every observed depth endpoint. Joinable
+book-ticker rows provide an independent check of top-of-book price and
+quantity. Every reconnect starts a separate snapshot-anchored segment.
+
+## Results
+
+| Measure | BTCUSDT | ETHUSDT |
+|---|---:|---:|
+| Snapshot-anchored segments | 4 | 3 |
+| Depth endpoints reconstructed | 1,668,215 | 1,659,917 |
+| Executable individual trades aligned | 2,847,061 | 4,498,386 |
+| Broken depth segments | 0 | 0 |
+| Late trades after validated buffer | 0 | 0 |
+| Exact book-ticker matches | 44,277 / 44,277 | 72,104 / 72,104 |
+| Zero-value `@trade` messages excluded | 11,850 | 12,741 |
+| Boundary-unassigned trades | 2,052 | 91 |
+| Observed trade quantity | 158,708.331 BTC | 4,524,196.266 ETH |
+| Required within-window replenishment | 17,306.673 BTC | 662,705.196 ETH |
+| Replenishment / trade quantity | 10.90% | 14.65% |
+
+The book-ticker denominator includes only rows whose update ID exactly matches
+a processed depth endpoint. It is not the count of all captured book-ticker
+messages.
+
+Zero-price, zero-quantity `@trade` messages are preserved in raw capture. Their
+raw payload includes undocumented fields such as `X=NA`; the study counts them
+but does not assign execution semantics or emit invalid zero-sized MBO rows.
+
+## Interpretation
+
+The results support the exchange-time alignment and minimum-flow accounting:
+
+- all depth segments remained sequence-continuous;
+- no executable trade arrived too late for the validated alignment buffer;
+- every independently joinable top-of-book state matched exactly;
+- the required replenishment ratio is measurable rather than hidden.
+
+They do not identify the true order-level queue. Both named policies can match
+the same L2 endpoints while producing different queue-ahead paths for a
+hypothetical resting order. Strategy conclusions that change materially
+between the two policies should be reported as model-sensitive.
+
+## Reproduce
+
+Run the study against a completed raw capture directory:
+
+```bash
+ordersim-binance-reconstruction-study captures/binance \
+  --symbol BTCUSDT \
+  --quantity-step 0.001 \
+  --reorder-buffer-ms 60000 \
+  --until-received-at-ns 1786381977000000000 \
+  --output reports/btcusdt-reconstruction.json
+```
+
+The JSON report is a model manifest: it records the policy names, quantity
+unit, cutoff, segment identifiers, alignment exceptions, inferred flow, and
+book-ticker checks.
+
+## Open Validation Questions
+
+Useful contributions include:
+
+- cancellation-allocation models supported by published microstructure work;
+- comparisons against private MBO for a venue or interval where both L2 and L3
+  are available;
+- passive-fill sensitivity experiments across the conservative and optimistic
+  bounds;
+- evidence about Binance's zero-value `@trade` messages without relying on
+  undocumented fields as stable production contracts.

docs/connectors.md

@@ -126,9 +126,8 @@ any non-consecutive trade ID observed within one individual-trade connection.
 
 These files are intentionally not canonical replay data. Binance depth has no
 stable public order IDs, and individual additions and cancellations inside an
-update window are not observable. A named L2-to-virtual-L3 model must consume
-the typed capture records, document the inference policy, and only then emit
-modeled `MBOEvent` rows.
+update window are not observable. `BinanceMBOReconstructor` consumes aligned
+typed records under a named policy and only then emits modeled `MBOEvent` rows.
 
 Capture files are local research data and must not be committed to the
 repository.
@@ -222,6 +221,51 @@ Aggregate trades preserve Binance's optional `nq` field as
 `normal_quantity`. When present, it is the quantity excluding trades involving
 RPI orders. When absent, `normal_quantity` is `None`, not an inferred value.
 
+### Studying L2-to-Virtual-MBO Reconstruction
+
+Run the evidence study before materializing modeled MBO for a new symbol or
+capture:
+
+```bash
+ordersim-binance-reconstruction-study captures/binance \
+  --symbol BTCUSDT \
+  --quantity-step 0.001 \
+  --output reports/btcusdt-reconstruction.json
+```
+
+The study streams raw files in receive order and aligns individual trades to
+depth intervals using Binance transaction timestamps. A configurable
+receive-time buffer permits messages from the independent depth and trade
+connections to arrive in a different local order. The default is 60 seconds,
+chosen from the public validation capture; every report includes maximum trade
+receive delay and residual late-trade lag so another environment can verify the
+margin. It reports:
+
+- broken or stale depth segments;
+- trades arriving too late for their exchange-time interval;
+- maximum executable-trade receive delay;
+- zero-price, zero-quantity `@trade` messages excluded from execution flow;
+- minimally inferred add and cancel quantities;
+- visible replenishment required to support observed trades;
+- exact book-ticker matches where update IDs can be joined;
+- metrics for both named queue policies.
+
+`queue-conservative` is the recommended default for fill research because it
+does not choose the cancellation and addition assumptions most favorable to a
+resting strategy. `queue-optimistic` is a sensitivity bound. If the strategy's
+result changes materially between them, the capture does not support one
+precise queue-fill claim.
+
+The minimum-flow identity preserves every observed L2 endpoint. Endpoint
+equality therefore validates implementation, not the hidden queue hypothesis.
+Book-ticker joins, sequence continuity, late-trade counts, and required
+replenishment provide the independent evidence in the report.
+
+Each fresh REST snapshot begins a new reconstruction segment. Do not concatenate
+segments as if the reconnect were an observed continuous MBO stream; write
+separate canonical files or preserve an explicit segment boundary in the
+research workflow.
+
 For the user-facing decision guide, see `docs/data-guide.md`.
 
 ## In-Memory Sources

docs/data-guide.md

@@ -111,12 +111,19 @@ snapshots, sequence-validated depth updates, trades, and book tickers. The main
 capture preserves Binance's individual `@trade` stream. Run
 `ordersim-binance-raw-trades` beside it for REST reconciliation and RPI trade
 flags; retain `aggTrade` only as another reconciliation feed. That typed source
-is the input boundary for the planned named virtual-L3 reconstruction model:
+is the input boundary for the named virtual-L3 reconstruction model:
 
 ```text
 raw capture -> BinanceCaptureSource -> named model -> modeled MBO + manifest
 ```
 
+Use `BinanceMBOReconstructor` to produce canonical `MBOEvent` rows for one
+snapshot-anchored connection segment. Use
+`ordersim-binance-reconstruction-study` first on a new capture or symbol to
+measure alignment, inferred flow, required replenishment, and the difference
+between the named queue policies. Reconnect segments remain separate because
+canonical MBO has no implicit clear-book event.
+
 There is deliberately no direct
 `BinanceCaptureSource -> Replay` path. See `docs/connectors.md` for the capture
 and validation contract.

docs/schema.md

@@ -42,10 +42,15 @@ event and transaction/trade times normalized from Binance milliseconds to UTC
 nanoseconds.
 
 Binance contract quantity can be fractional, so the connector preserves it as
-`Decimal`. A future virtual-L3 reconstruction model must declare its quantity
-unit and exact conversion rule before producing the canonical integer
-`MBOEvent.size`. These L2 records are therefore not accepted directly by
-`Replay`.
+`Decimal`. `BinanceReconstructionConfig.quantity_step` declares the exact unit
+used to produce canonical integer `MBOEvent.size` values. A quantity that is
+not exactly divisible by that unit is rejected rather than rounded. The typed
+L2 records are not accepted directly by `Replay`.
+
+The real-time `@trade` stream can contain zero-price, zero-quantity messages
+whose untouched raw payload reports `X=NA`. The reconstruction study counts
+these separately as `zero_value_trade_messages`; it does not present them as
+executions or emit zero-sized canonical rows.
 
 Raw-trade capture files also contain audit envelopes:
 

pyproject.toml

@@ -67,6 +67,7 @@ parquet = [
 [project.scripts]
 ordersim-binance-capture = "ordersim.connectors.binance.capture:main"
 ordersim-binance-raw-trades = "ordersim.connectors.binance.raw_trades:main"
+ordersim-binance-reconstruction-study = "ordersim.connectors.binance.reconstruction_study:main"
 
 [project.urls]
 Repository = "https://github.com/tradingexpert/ordersim"

src/ordersim/connectors/binance/__init__.py

@@ -13,11 +13,19 @@
     BinanceDepthSnapshot,
     BinanceDepthUpdate,
     BinanceIndividualTrade,
+    BinanceObservedEvent,
     BinancePriceLevel,
     BinanceRawTrade,
     DepthStreamKind,
 )
 from ordersim.connectors.binance.raw_trades import capture_binance_raw_trades
+from ordersim.connectors.binance.reconstruction import (
+    BinanceMBOReconstructor,
+    BinanceReconstructionConfig,
+    BinanceReconstructionMetrics,
+    BinanceReconstructionPolicy,
+    BinanceReconstructionStep,
+)
 from ordersim.connectors.binance.schema import (
     BinanceCaptureConfig,
     BinanceRawTradeCaptureConfig,
@@ -38,9 +46,15 @@
     "BinanceDepthSnapshot",
     "BinanceDepthUpdate",
     "BinanceIndividualTrade",
+    "BinanceMBOReconstructor",
+    "BinanceObservedEvent",
     "BinancePriceLevel",
     "BinanceRawTrade",
     "BinanceRawTradeCaptureConfig",
+    "BinanceReconstructionConfig",
+    "BinanceReconstructionMetrics",
+    "BinanceReconstructionPolicy",
+    "BinanceReconstructionStep",
     "BinanceSequenceError",
     "CaptureManifest",
     "DepthStreamKind",

src/ordersim/connectors/binance/l2.py

@@ -151,3 +151,9 @@ class BinanceBookTicker:
 
 
 BinanceDepthEvent: TypeAlias = BinanceDepthSnapshot | BinanceDepthUpdate
+BinanceObservedEvent: TypeAlias = (
+    BinanceDepthSnapshot
+    | BinanceDepthUpdate
+    | BinanceIndividualTrade
+    | BinanceBookTicker
+)