Microstructure alpha: signal construction and honest validation

Order book features built from raw NASDAQ Level 3 message data, and a validation framework that measures how much of a backtested edge is real.

The features are the usual microstructure suspects — order book imbalance, trade flow imbalance, VPIN, microprice deviation, short-horizon return autocorrelation — computed from LOBSTER message and book files for AMZN on 2012-06-21, 269,748 events over a full session.

The finding is not about the features.

pip install -r requirements.txt
bash scripts/fetch_data.sh                  # LOBSTER sample, ~74 MB
python scripts/run_study.py --build --all   # full study, ~5 min
pytest -q                                   # 14 tests

---

The result

Two validation procedures, same features, same model, same data. The only difference is how the train/test split is formed.

forward horizon	shuffled k-fold	purged walk-forward	inflation	rows	effective n
1s	0.6264	0.6127	+0.0137	6,370	6,370
5s	0.7257	0.5729	+0.1528	14,860	2,972
10s	0.7567	0.5382	+0.2185	18,171	1,817
30s	0.8160	0.5144	+0.3016	20,749	692
60s	0.8300	0.4981	+0.3319	21,007	350

An AUC of 0.83 at a one-minute horizon would be a remarkable result. It is entirely an artefact. The honest figure at the same horizon is 0.498 — a coin flip.

What makes this worth showing is the direction of the two curves. The naive score improves as the horizon lengthens; the real one decays to nothing. Both are driven by the same mechanism. A 60-second forward return sampled every second means adjacent labels share 59 of their 60 seconds, so a shuffled split puts near-duplicates of the test labels into the training set. The longer the horizon, the more overlap, the more leakage — and simultaneously the fewer independent observations the data actually contains. The number that looks best comes from the setup with the least real information in it.

This is the failure mode that makes a strategy look excellent in research and lose money in production, and it requires no incompetence to produce: shuffled k-fold is the default in every ML tutorial.

Where the signal actually is

Per-feature univariate AUC, tested against a circular block bootstrap null that preserves the autocorrelation of the label series, with Benjamini-Hochberg control across the 15 features.

1-second horizon — 9 of 15 features survive

feature	AUC	p (block bootstrap)	p adjusted	significant
tfi_5s	0.5882	0.000	0.000	yes
obi_1	0.5842	0.000	0.000	yes
microprice_dev	0.5781	0.000	0.000	yes
tfi_30s	0.5778	0.000	0.000	yes
ret_30s	0.5582	0.000	0.000	yes
ret_5s	0.5578	0.000	0.000	yes
ret_1s	0.5356	0.000	0.000	yes
obi_5	0.5244	0.001	0.002	yes
depth_ratio	0.5244	0.001	0.002	yes
ret_ac1	0.5064	0.469	0.662	no
obi_10	0.4937	0.493	0.662	no
spread_rel	0.4952	0.551	0.662	no
vpin	0.4952	0.573	0.662	no
trade_rate_30s	0.4965	0.704	0.754	no
rv_30s	0.5017	0.841	0.841	no

30-second horizon — 1 of 15 survives

feature	AUC	p	p adjusted	significant
tfi_5s	0.5428	0.000	0.000	yes
ret_30s	0.4705	0.051	0.385	no
ret_1s	0.5060	0.122	0.610	no
tfi_30s	0.5169	0.251	0.943	no
(11 more, all p adjusted > 0.9)				no

At one second, top-of-book imbalance and recent signed trade flow genuinely predict the next mid-price move. That is not a surprise — it is what the microstructure literature says should happen, and recovering it is evidence the pipeline works rather than a discovery. By thirty seconds, order book imbalance is at 0.502 and microprice deviation at 0.496. The information has been priced in.

Note ret_30s at the 30-second horizon: raw p of 0.051, adjusted p of 0.385. Screening fifteen features and reporting the best raw p-value would have turned that into a finding. It is the single clearest argument for correcting across the whole feature set rather than the one that looked promising.

A bug worth describing

VPIN sizes its volume buckets as a fraction of daily volume. The first implementation here computed that fraction from the session's total traded volume — which is not observable intraday. The second attempt scaled a warm-up window up to the full session length, which still used the closing time.

Neither was visible by inspection. The feature's distribution looked entirely plausible in both cases. What caught it was a test that rebuilds every feature from a truncated copy of the session and requires bit-identical values on the overlap: if anything at time t depends on data after t, truncation moves it.

tests/test_alpha.py::test_features_do_not_use_future_data is the most valuable test in the suite, and it exists because this class of error does not announce itself.

Method

Features. Computed on a one-second grid from raw L3 messages. Every value at time t uses a strictly trailing window. Trade signing needs no tick rule — LOBSTER identifies the resting side of every execution, so an execution against a resting sell is by definition buyer-initiated.

Purged walk-forward CV. Sequential splits only; a model that trains on the afternoon to predict the morning is not tradeable. Training rows whose label window reaches into the test period are dropped, plus an embargo of twice the horizon to absorb feature autocorrelation running the other way.

Effective sample size. Reported alongside row count throughout. 20,749 rows at a 30-second horizon is 692 independent observations. Confidence intervals computed on the former are roughly five times too narrow.

Block bootstrap null. Features and labels are both strongly autocorrelated, so an AUC of 0.53 arises by chance far more often than an i.i.d. null implies. Circular block resampling preserves that structure. The test suite includes a check that this null is genuinely wider than a permutation null when both series are autocorrelated — and it is, by more than a factor of two.

Benjamini-Hochberg. Controls the expected proportion of false discoveries among rejections. Bonferroni controls the probability of any false positive at all, which is the wrong guarantee when screening candidate features and strict enough to reject everything.

Layout

alpha/
  features.py     LOBSTER loading, microstructure features, forward labels
  validation.py   purged walk-forward, effective n, block bootstrap, BH-FDR
  model.py        fixed XGBoost configuration
scripts/
  run_study.py    reproduces every table and figure above
  fetch_data.sh   downloads the LOBSTER sample
tests/
  test_alpha.py   14 tests

Limitations

These matter more than usual here, because the project is about not overstating results.

• One stock, one session. Six and a half hours of AMZN in June 2012. The 1-second findings are consistent with the literature, but nothing here establishes that they generalise to another stock, another day, or another decade. LOBSTER publishes a single free sample day.
• 692 independent observations at 30 seconds. Enough to demonstrate that a signal is absent; nowhere near enough to establish that one is present.
• The book is truncated at 10 levels. LOBSTER's message file only reports events affecting the top ten, so obi_10 in particular is computed from a book with unobservable depth beneath it.
• No transaction costs. AUC measures directional accuracy, not profit. The 1-second edge sits well inside a 13-cent median spread, so it is not by itself tradeable — capturing it would require queue position and passive fills, which this data cannot model.
• VPIN is a within-day variant. The original uses a fifty-bucket window spanning several days. On a single session that window never fills, so buckets are sized to give roughly fifty per warm-up window instead.
• One model configuration, no tuning. Deliberate: tuning per horizon would add another layer of selection over an already thin effective sample.

Using other data

The feature and validation code takes a dataframe, not a file format. To run it on crypto tick data, replace load_lobster with a loader producing the same columns — timestamp, trade size, price, trade sign, and top-of-book depth — and build_panel onward is unchanged. The validation machinery is entirely data-agnostic.

References

• Easley, López de Prado & O'Hara (2012), Flow Toxicity and Liquidity in a High-Frequency World
• López de Prado (2018), Advances in Financial Machine Learning — purging, embargo, sample uniqueness
• Benjamini & Hochberg (1995), Controlling the False Discovery Rate
• Politis & Romano (1994), The Stationary Bootstrap