Skip to content

get_data.md

Latest commit

 

History

History
49 lines (39 loc) · 2.37 KB

get_data.md

File metadata and controls

49 lines (39 loc) · 2.37 KB
Raw

LOBSTER sample data

The project runs without external data using the seeded synthetic source in config/config.example.yaml. Synthetic data is pipeline-test data and must remain labelled SIMULATED. Real microstructure analysis (Module A, including the cross-section) needs genuine LOBSTER files.

Official sample

The analysis uses the public sample symbols AAPL, AMZN, GOOG, INTC, and MSFT on 2012-06-21 at level 1, plus the matching AAPL level-5 pair. Obtain sample data from the official LOBSTER sample page or another source you are authorized to use.

The code is MIT-licensed. LOBSTER data is not included and is governed by separate terms. Do not commit or redistribute the raw files through this repository.

Expected files

data/raw/AAPL_2012-06-21_34200000_57600000_message_1.csv
data/raw/AAPL_2012-06-21_34200000_57600000_orderbook_1.csv
data/raw/AMZN_2012-06-21_34200000_57600000_message_1.csv
data/raw/AMZN_2012-06-21_34200000_57600000_orderbook_1.csv
data/raw/GOOG_2012-06-21_34200000_57600000_message_1.csv
data/raw/GOOG_2012-06-21_34200000_57600000_orderbook_1.csv
data/raw/INTC_2012-06-21_34200000_57600000_message_1.csv
data/raw/INTC_2012-06-21_34200000_57600000_orderbook_1.csv
data/raw/MSFT_2012-06-21_34200000_57600000_message_1.csv
data/raw/MSFT_2012-06-21_34200000_57600000_orderbook_1.csv
data/raw/AAPL_2012-06-21_34200000_57600000_message_5.csv
data/raw/AAPL_2012-06-21_34200000_57600000_orderbook_5.csv

After fetching, the level-1 loader validates equal row counts and the epsilon = -direction trade-sign sanity check. The impact audit separately validates L5 row counts, chronology, and L1/L5 top-state alignment. Do not add headers.

Configuration

  • config.example.yaml uses the synthetic source and has a commented cross_section block.
  • The local, git-ignored config/config.yaml sets data.source: lobster, data.label: REAL, points message_path/orderbook_path at AAPL (Module B), and lists all five names under cross_section.symbols (Module A). impact_audit enables the REAL-vs-SIMULATED headline audit and points to the AAPL level-5 pair. The synthetic cold-clone path skips it.

Message and order-book files must cover the same symbol, date, time range, and book depth, and have the same number of rows. Raw files are git-ignored and must not be committed or redistributed.