feat: add deterministic normalized v3 and fail-closed L2 capture

.gitattributes

@@ -0,0 +1,2 @@
+tests/fixtures/providers/*.json text eol=lf
+tests/fixtures/providers/**/*.json text eol=lf

.gitignore

@@ -3,8 +3,12 @@
 __pycache__/
 *.py[cod]
 .venv/
+.venv-m7-verify/
+.m7/
+validation/performance/work-*/
 .pytest_cache/
 .coverage
+coverage.json
 *.egg-info/
 data/
 dist/

README.md

@@ -77,11 +77,19 @@ move or rebuild a release tag to repair a dependency resolution.
 | `curated` | Session-aware bar aggregation and immutable revision/lineage snapshots |
 | `l2_replay` | Deterministic Snapshot+Delta reconstruction, sequence/cross checks and checkpoint hashes |
 | `adapters_v2` | Binance, OKX and supplier-neutral domestic desensitized fixture adapters |
+| `capture_v2` | Fail-closed public Binance/OKX L2 capture, immutable batched Raw segments, snapshot synchronization, independent archive/restore verification and the Raw-to-Normalized bridge |
+
+The M7 Arrow batch entrypoint, bounded-memory validation architecture, benchmark scope, and
+the explicit gap between frozen fixtures and current OKX/Binance live-book semantics are documented
+in [`docs/m7-data-performance.md`](docs/m7-data-performance.md).
+The public-feed collector's exact eight-stream scope, explicit storage configuration, state machine,
+safe CLI modes and non-certification boundary are documented in
+[`docs/m7-crypto-l2-capture.md`](docs/m7-crypto-l2-capture.md).
 
 ## M2 data-lake guarantees
 
 - Raw persists exact provider bytes through lake-local staging and atomic rename. Its integrity anchor binds source, request, UTC collection time, object/key path identity, SHA-256 and the30-day retention policy. A crash-released process lock and immutable key claim make concurrent writes, crash recovery and cleanup serialize on the same idempotency key. Every write, read and cleanup rejects path escapes and Windows reparse points below the lake root.
-- Normalized requires resolvable, hash-verified Raw references and writes only frozen`standard/v2`Arrow schemas under`provider/venue/event_type/date/instrument`partitions. A sharded persistent claim index binds every lake-wide`event_id`to its Arrow-normalized logical event hash. Same-ID/same-content reuse is idempotent; conflicting content, bad sequences and L2 reconstruction failures cannot enter research snapshots.
+- Normalized requires resolvable, hash-verified Raw references and writes only frozen`standard/v2`Arrow schemas under`provider/venue/event_type/date/instrument`partitions. Capture epochs persist`PREPARED`before snapshot publication and finish as`COMMITTED`or`ABORTED`; startup uses the frozen stream configuration as an independent identity anchor, enforces closed terminal JSON fields and strict types, recomputes partition rows, logical hashes, the available-time maximum and the final L2 state from the immutable journal, and rejects any receipt bound to a different snapshot before network startup. A sharded persistent claim index binds every lake-wide`event_id`to its Arrow-normalized logical event hash. Same-ID/same-content reuse is idempotent; conflicting content, bad sequences and L2 reconstruction failures cannot enter research snapshots.
 - The certified Curated entry is`curate_trade_bars_from_snapshot`: it reads trades from one explicit verified Normalized snapshot, constructs session-aware bars and binds the exact lineage. One`dataset+revision_id`maps to one snapshot; corrected data requires a new revision and never overwrites history.
 - Normalized and Curated snapshot identities bind Arrow-canonical logical rows and physical Parquet hashes. DuckDB verifies the fixed snapshot, copies Arrow data into in-memory tables, then disables external access; user SQL cannot call file readers or resolve`latest`/`main`.
 - Collection stops with a visible`COLLECTION_STOPPED`error if hot data would exceed150GB or free space would fall below`max(volume*20%,100GB)`.
@@ -90,7 +98,7 @@ move or rebuild a release tag to repair a dependency resolution.
 
 ## M2 fixture certification scope
 
-For both Binance and OKX, the certified fixture set is deliberately narrow: BTC spot covers Trade、BBO、BookSnapshot和BookDelta; ETH spot covers Trade only; BTC and ETH perpetuals cover FundingRate and MarkPrice only. It does not claim every event type for all four instruments. OKX books use the provider's signed CRC32 checksum; Binance has no equivalent field, so its gate is U/u/pu continuity plus immutable Raw SHA-256. The domestic supplier-neutral L2 fixture is deliberately marked`fixture-certified-not-market-data-certified`; it must not be described as real domestic market-data certification.
+For both Binance and OKX, the certified fixture set is deliberately narrow: BTC spot covers Trade、BBO、BookSnapshot和BookDelta; ETH spot covers Trade only; BTC and ETH perpetuals cover FundingRate and MarkPrice only. It does not claim every event type for all four instruments. The bundled OKX nonzero signed-CRC32 book sample is a historical golden fixture only: since 2026-06-23 the live JSON field is fixed to0 and is not an integrity gate. A current OKX collector must use TLS, enforce`seqId/prevSeqId`continuity, consume empty equal-sequence heartbeats without emitting`BookDelta`, and terminate admission on maintenance resets until a fresh snapshot. The separate`capture_v2`module can open only public Binance/OKX market-data endpoints after physical-volume, capacity and archive/restore preflight succeeds. Its deterministic tests certify the transport and synchronization implementation, not a continuous real-market dataset. Binance has no equivalent checksum field, so its fixture gate is U/u/pu continuity plus immutable Raw SHA-256. The domestic supplier-neutral L2 fixture is deliberately marked`fixture-certified-not-market-data-certified`; it must not be described as real domestic market-data certification.
 
 ## PIT rules
 
@@ -106,6 +114,7 @@ qdk-validate data/prices.parquet
 qdk-manifest data/prices.manifest.json
 qdk-catalog list
 qdk-catalog register hs300_prices data/prices.parquet
+qdk-capture capture.json --mode preflight
 ```
 
 ## Python API

docs/m7-crypto-l2-capture.md

@@ -0,0 +1,149 @@
+# M7 public Crypto L2 capture
+
+## Frozen scope and safety boundary
+
+`quant_data_kit.capture_v2`collects public market data only. It has no credential, account or order
+API, rejects credential-like configuration keys recursively, and sends no orders. The default and
+machine-certified scope is exactly these eight streams:
+
+| Provider | Market | Instruments | Feed |
+|---|---|---|---|
+| Binance | Spot | BTCUSDT、ETHUSDT | Public depth WebSocket plus public REST depth snapshot |
+| Binance | USDT linear perpetual | BTCUSDT、ETHUSDT | Public USD-M depth WebSocket plus public REST depth snapshot |
+| OKX | Spot | BTC-USDT、ETH-USDT | Public`books`WebSocket |
+| OKX | USDT perpetual | BTC-USDT-SWAP、ETH-USDT-SWAP | Public`books`WebSocket |
+
+The run report always emits sorted providers`["binance","okx"]`and the sorted capabilities
+`btc-spot-l2`、`btc-usdt-perpetual-l2`、`eth-spot-l2`and
+`eth-usdt-perpetual-l2`. Configuration may list the streams explicitly, but M7 rejects any change
+to provider、market、native symbol、stable instrument、stream ID、venue、channel or the parsed
+official WSS/HTTPS host、port、path and query identity.
+
+This implementation is not evidence of30 continuous capture days and is not
+`market-data-certified`. A bounded probe proves only that preflight, public transport and admission
+work for that invocation.
+
+## State and synchronization rules
+
+Every stream has the explicit state sequence`CONNECTING→BUFFERING→SNAPSHOT_SYNC→LIVE`and can
+terminate in`RESYNC`or`PAUSED`. State transitions、failures and retries are committed first to a
+dedicated content-addressed durable audit chain; only a successful reload/hash check allows the
+in-memory state to change. Illegal transitions are durably audited and then rejected. High-rate
+book observations remain counted, with the first and each1024th occurrence durably summarized so
+that operational audit does not become per-market-message fsync I/O.
+
+Binance messages are buffered before requesting the HTTPS snapshot. The bridge discards stale
+updates, requires the first admitted update to satisfy`U<=lastUpdateId<=u`, then enforces the USD-M
+`pu`chain or the applicable Spot update-ID continuity. Quantities are absolute; zero deletes a
+level. Deleting an absent level is retained in Raw and audited as a documented no-op, not reported
+as book corruption.
+
+OKX requires TLS and a fresh`action=snapshot`before updates. It enforces
+`seqId/prevSeqId`continuity. A zero checksum is not integrity evidence, an equal-sequence empty
+heartbeat creates no Normalized event, and a maintenance reset forces`RESYNC`and blocks deltas
+until a new snapshot.
+
+## Immutable Raw and existing Normalized contract
+
+Each WebSocket or HTTPS response is preserved byte-for-byte in one`RawFrame`; frames are batched by
+message count, exact wire-byte count or age into immutable Raw segments. A segment binds provider,
+stream, receive/event time ranges, native sequence ranges, message count, wire bytes, SHA-256,
+collector commit and previous-segment lineage. A segment is never one file per network message and
+an existing immutable path cannot be overwritten with different bytes.
+
+Admitted snapshots and deltas are spooled in bounded Normalized epochs. Publication uses the
+existing strict Arrow-batch path, PIT checks, schemas, quarantine rules and Raw references. Every
+finalize attempt persists an immutable`PREPARED`transaction before publishing a visible snapshot,
+then ends in a content-addressed`COMMITTED`receipt or`ABORTED`failure. Before any network startup,
+the coordinator anchors every journal to the frozen stream configuration, scans all journal
+parents, enforces closed JSON fields, strict types and filename attempt binding for every terminal
+record, then recomputes partition rows, logical hashes, the available-time maximum and the final L2
+state from the immutable journal. A receipt that points to another individually valid snapshot is
+rejected before network startup. Unresolved`PREPARED`or retryable`ABORTED`transactions replay
+idempotently.
+A real spawned-process test terminates with`os._exit`after snapshot publication but before receipt
+publication and proves that a fresh process commits the transaction before any network runner is
+created. A transaction without a durable PREPARED identity blocks startup instead of being skipped.
+Gaps or connection failures explicitly abort the current epoch before resynchronization.
+
+## Capacity and independent archive controls
+
+`hot_root`、`archive_root`and`restore_root`must be explicit absolute existing directories. The hot
+and archive roots are resolved through an injectable physical-volume probe; different path strings
+on the same physical device fail preflight. All three roots and each final parent reject symbolic
+links、Windows junctions/reparse points and resolved escapes. The bootstrap lock itself is created
+and checked as a regular non-reparse object before the file-lock backend may open it. These rules
+apply to journal creation, parts, transaction recovery, receipts and abort records as well as
+Raw/archive paths. Collection takes an exact startup
+baseline, reserves every projected write under a thread-safe incremental counter, and repeats real
+tree/capacity probes at bounded message、byte or monotonic-time intervals. It pauses if projected
+hot data exceeds150GiB or if free space falls below
+`max(volume capacity*20%,100GiB)`. Archive reserve failures also pause collection. No capture path
+deletes or evicts data automatically.
+
+Preflight writes an immutable archive probe, copies it to the archive volume, restores it under the
+explicit restore root and recomputes SHA-256. Every Raw segment then repeats copy, manifest and
+payload hashing, temporary restore, model re-read and hash comparison. The destination physical
+identity is rechecked before and after immutable create-if-absent publication. Only a successful receipt
+sets`archive_restore_verified=true`and`eligible_for_cleanup=true`; it always records
+`cleanup_performed=false`. Cleanup is outside this collector and requires a separate explicit
+operator action.
+
+## Configuration and CLI
+
+Example configuration, with operator-selected absolute paths:
+
+```json
+{
+  "hot_root": "<absolute-hot-directory>",
+  "archive_root": "<absolute-archive-directory-on-a-different-physical-volume>",
+  "restore_root": "<absolute-temporary-restore-parent>",
+  "collector_commit": "<git-commit>",
+  "archive_reserve_bytes": 161061273600,
+  "rotation": {
+    "max_messages": 2000,
+    "max_wire_bytes": 16777216,
+    "max_age_seconds": 30
+  },
+  "retry": {
+    "max_attempts": 5,
+    "base_delay_seconds": 0.5,
+    "maximum_delay_seconds": 8.0,
+    "jitter_fraction": 0.2
+  },
+  "durability": {
+    "capacity_probe_messages": 256,
+    "capacity_probe_bytes": 4194304,
+    "capacity_probe_seconds": 1.0,
+    "normalized_flush_records": 256,
+    "normalized_flush_bytes": 1048576,
+    "normalized_flush_seconds": 1.0,
+    "probe_timeout_seconds": 30.0
+  }
+}
+```
+
+Safe preflight is the default and opens no network connection:
+
+```bash
+qdk-capture capture.json
+qdk-capture capture.json --mode preflight
+```
+
+A bounded public probe is explicit and still runs archive preflight first. Its message budget covers
+the whole post-connect synchronization phase, including OKX control frames; both timeout and message
+budget exhaustion are audited failures:
+
+```bash
+qdk-capture capture.json --mode probe --max-messages 3
+```
+
+Unbounded collection additionally requires`--confirm-long-running`. It still refuses to connect if
+physical independence, free-space or archive/restore verification fails:
+
+```bash
+qdk-capture capture.json --mode run --confirm-long-running
+```
+
+Do not use`run`as a service readiness claim. Operational certification still requires retained
+continuous evidence, data-quality review and an independently reviewed capacity/retention plan.

docs/m7-data-performance.md

@@ -0,0 +1,101 @@
+# M7 Arrow normalization and market-feed boundary
+
+## Public entrypoints
+
+- `write_normalized_events(...)` remains the compatibility API for
+  `Iterable[Mapping]`. It preserves stream-level quarantine behavior.
+- `write_normalized_batches(...)` accepts `Iterable[pyarrow.RecordBatch]` or a
+  `pyarrow.RecordBatchReader`. Every non-empty batch must have the exact frozen
+  v2 Arrow schema and be homogeneous by event type, instrument, and trading day.
+  Invalid batch input fails the normalization transaction closed; it does not
+  publish a partial snapshot or a quarantine object.
+- The fast L2 path is limited to a vector-proven sequence of same-side,
+  same-price upserts. Any delete, mixed price/side/action, requested interior
+  checkpoint, or non-uniform stream uses the serial reconstructor.
+
+Both entrypoints enforce trusted Raw references, schema and PIT rules, strict
+sequence continuity, deterministic L2 reconstruction, duplicate event IDs,
+lake-wide event-claim conflicts, capacity stops, immutable publication, and
+post-publication verification.
+
+## Versioned storage algorithms
+
+Normalized layout `3.0.0` records these independently versioned algorithms:
+
+- Mapping partitions use `canonical-json-array-v1`, preserving legacy logical
+  partition hashes.
+- Arrow partitions use `arrow-ipc-record-batch-v1`. External loads re-read the
+  Parquet data and recompute this digest; the manifest never substitutes for
+  verification.
+- Claim indexes use `streaming-parquet-v2` with claim version `3.1.0`. Individual
+  `event_id_hash`, `event_sha256`, and `claim_sha256` values retain the frozen v2
+  canonical JSON definitions. A versioned, order-independent
+  `sha256-multiset-u64x4-v1` shard digest removes the former unbounded sorted
+  string aggregation. Physical file hashes and a manifest hash remain mandatory.
+
+Historical v2 snapshots are not rewritten. A missing v3 claim acceleration file
+can be rebuilt from immutable partitions; a changed file or manifest fails closed.
+
+## 2026 exchange-feed boundary
+
+The frozen normalized event contract remains independent of network transports. The M7 public-feed
+collector implements the following admission semantics in`capture_v2`and bridges admitted records
+into that existing contract; it does not create a second Normalized schema:
+
+- OKX deprecated the JSON `checksum` for `books`, `books-l2-tbt`, and
+  `books50-l2-tbt` on 2026-06-23. The retained field is zero and is not a current
+  integrity signal. The collector requires TLS`wss`transport and strict
+  `seqId`/`prevSeqId`processing. Sources:
+  [checksum deprecation](https://www.okx.com/en-us/help/okx-order-book-channels-checksum-field-deprecation)
+  and [OKX API v5](https://www.okx.com/docs-v5).
+- An OKX no-change heartbeat can have empty `asks`/`bids` and
+  `seqId == prevSeqId`. It is transport liveness, not a frozen `BookDelta`, and
+  does not enter the normalized writer. A maintenance reset with
+  `seqId < prevSeqId` must terminate the current admission transaction, discard
+  deltas until a fresh snapshot, and start a new normalization transaction.
+  The current v2 event contract intentionally rejects both equal-sequence deltas
+  and an in-transaction sequence rollback.
+- Binance USDⓈ-M local-book admission must first bridge
+  `U <= lastUpdateId <= u`, then require each event's `pu` to equal the prior `u`.
+  Quantities are absolute and zero means deletion. Binance documents that deleting
+  an absent local price can be normal. The collector audits that raw event as a
+  measured no-op before emitting normalized deltas, because the
+  frozen reconstructor rejects an absent-level delete. Source:
+  [Binance USDⓈ-M local order book](https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/How-to-manage-a-local-order-book-correctly).
+- Existing OKX CRC32 fixture tests remain historical fixture-integrity tests only.
+  They are not labeled as 2026 OKX market-data certification.
+
+The collector's exact scope, archive controls and safe CLI are documented in
+[`m7-crypto-l2-capture.md`](m7-crypto-l2-capture.md). These implementation tests do not establish
+continuous30-day evidence or real market-data certification, and they are not reasons to weaken
+the normalized batch gates.
+
+## Performance gate
+
+The formal command runs each repetition in a new Python process. Raw admission
+setup is excluded; RecordBatch generation and the complete normalized write are
+inside the timed scope.
+
+```powershell
+$env:TEMP = 'F:\puresaber-m7-temp'
+$env:TMP = 'F:\puresaber-m7-temp'
+python tools/benchmark_normalized_l2.py `
+  --work-root .m7\quant-data-kit-10m-final-clean2-009a361 `
+  --output validation/performance/m7-data-arrow-10m-final-okx-contract.json `
+  --rows 10000000 --runs 3 --batch-rows 262144 `
+  --minimum-events-per-second 100000 --maximum-peak-rss-gib 16
+```
+
+The report records the machine, Python and dependency versions, Git identity,
+timed scope, three run results, deterministic artifact fields, C/H/F disk space,
+actual temporary directory, and whether generated synthetic data was removed.
+The formal run retains all three lake directories under the recorded H-drive
+work root; cleanup is a separate, explicit operator decision after evidence review.
+
+The final clean run used commit`009a36162a2ec1a48fc4f96b93b2e675196e9263`with
+`dirty=false`. Its three throughputs were155,932.27、155,071.20 and153,097.30events/s;
+maximum peak RSS was2.967GiB. All30,000,000 rows were accepted, no row was quarantined,
+strict reload passed, and snapshot/partition/claim/L2-checkpoint hashes were identical across
+fresh processes. The three retained runs contain3,310,487,778bytes. `TEMP`, `TMP`, and Python's
+actual `tempfile.gettempdir()` all resolved to`F:\puresaber-m7-temp`; C was not the temp volume.
+The report SHA-256 is`69416eeba389ff520043c9382ed7e1ff7380f4d5937030a02cb84cb1ab80c08f`.