diff --git a/CHANGELOG.md b/CHANGELOG.md index 27ec43b10..674e59c89 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,12 @@ before 1.0). - **Lightning chain backends (`Q-69` Open):** [`docs/lightning.md`](docs/lightning.md) owns CLN `bcli` and ldk-node Esplora/Electrum. +- **Schema 25 `txstat.body`:** 8 B/create ULEB confirm-time econ + (`n_in`/`fee_sat`/`base`/`wit_extra`) with per-header remaining-byte overflow. + Occupied 24 open rewrites `meta` and zero-extends the file to loc count (no + `txout.body` rewrite). Unreleased leftover `txfixed.body` is unlinked. A 24 + binary refuses 25 `meta`. + - **Core functional `mempool_packages.py`:** inventory `run`. Verbose mempool `vsize` / ancestor-descendant size use Core ceil-vsize; `wtxid` is on both `getmempoolentry` and verbose `getrawmempool`; non-verbose txid list is diff --git a/COMPAT.md b/COMPAT.md index a3ceb91b5..3e8ea2318 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -112,7 +112,7 @@ Per-method notes, auth, and the shindex matrix live in | Method group | Status | |--------------|--------| | Control (`help`, `uptime`, `stop`, `getrpcinfo`, `echo`) | done (`syncwithvalidationinterfacequeue` omitted; functional proxy no-op for Core `sync_mempools`) | -| Blockchain (`getblockchaininfo`, `getblockcount`, `getbestblockhash`, `getblockhash`, `getblock`/`header`, `getdifficulty`, `getblockstats`) | done (archive reconstruct; disk/progress real) | +| Blockchain (`getblockchaininfo`, `getblockcount`, `getbestblockhash`, `getblockhash`, `getblock`/`header`, `getdifficulty`, `getblockstats`) | done (`getblockstats` from `txstat` when stamped; omit coins-DB `utxo_size_*`) | | Network (`getnetworkinfo`, `getconnectioncount`, `getpeerinfo`, `addnode`, `disconnectnode`, `addconnection`) | done (BIP324 v2-only; peer `timeoffset` / `synced_*` from session state) | | Mempool / rawtx (`getmempool*`, `getrawtransaction`, `sendrawtransaction`, `testmempoolaccept`) | done (Libre; RPC `maxfeerate` / `maxburnamount` / `"version"` only) | | Coin / MiniWallet (`gettxout`, `scantxoutset` `raw(HEX)`) | done (Class A unspent walk — not a coins-DB) | @@ -223,7 +223,7 @@ App `ServeLimits` always on (same model as Electrum). | Tip | done | `/blocks/tip/height`, `/blocks/tip/hash`. REST stamps `X-Bitcoin-Chain-Tip` / `X-Bitcoin-Chain-Tip-Height` (CORS-exposed): **live tip** for block/tx/header routes; **SH watermark** for `/address/` and `/scripthash/` so wallet JSON matches the SH join. Empty chain omits them (existing 503). If the pin dies mid-request: **503** `chain view moved`. | | Blocks list | done | `/blocks`, `/blocks/:start_height` (10 summaries, newest-first) | | Block | done | `/block/:hash` JSON, `/raw`, `/status`, `/header`, `/txids`, `/txid/:i`, `/txs[/:start]`. JSON `bits` is the compact-target **u32** (Esplora schema, not Core hex). `size` / `weight` are BIP144 total size and BIP141 weight (witness included). | -| Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` (`vin` from the spent slot; unspent omits it). Mempool-only txs (not in Class A) use the wire body from the mempool hub (`vin`/`vout`/`size`/`weight`/`fee`/`sigops`, `status.confirmed` false) including `GET /tx/:txid/status`. Live `/outspend(s)` overlay mempool spends of confirmed coins; `?asof=` omits mempool. `?asof=` on `/status` and `/outspend(s)`: confirmed/spent as of that ancestor; 404 if not on chain. | +| Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` (`vin` from the spent slot; unspent omits it). Below inwit prune: JSON is 200 partial (`pruned: true`, vout+status, empty `vin`); stamped `txstat` fills `fee`/`size`/`weight`. Mempool-only txs (not in Class A) use the wire body from the mempool hub (`vin`/`vout`/`size`/`weight`/`fee`/`sigops`, `status.confirmed` false) including `GET /tx/:txid/status`. Live `/outspend(s)` overlay mempool spends of confirmed coins; `?asof=` omits mempool. `?asof=` on `/status` and `/outspend(s)`: confirmed/spent as of that ancestor; 404 if not on chain. | | Address / scripthash | done | stats + `/utxo` + `/txs` + `/txs/mempool` + `/txs/chain[/:last_seen_txid]` + `/txs/summary[/:last_seen_txid]` (dialect; next row). `/utxo` matches Electrum listunspent (mempool funding + drop mempool-spent confirmed); `/txs` and `/txs/mempool` use full Esplora tx JSON for mempool-only rows (wire from the hub). `?after_txid=` on `/txs` and `/txs/summary` skips through that tx (mempool then chain); unknown or unparseable → **422** `after_txid not found`. `POST /addresses/txs` and `POST /scripthashes/txs` (and `/txs/summary`) take a JSON array (max **300**; over → **422** `body too long`); unique scripts join once per request, merge newest-first. Unix/loopback `X-Rbitcoin-Client`: last-1 GET join + last-bulk POST (16 MiB packed/client), 30s idle, 256 clients, singleflight on the same `(client, sh)`. Public TCP ignores the header (stack-local join). Concurrent different GET scripts replace last-1. Unbounded process LRU stays **X-M3**. Needs SH finalize. Stamp is visible SH (durable + pending write-behind), matching live tip while jobs sit in RAM. `?asof=` on `/`, `/utxo`, `/txs`, `/txs/chain`, `/txs/summary`: confirmed join at that ancestor **at or behind visible SH**, **no** mempool; headers are the asof hash; 404 if not on chain or ahead of visible SH. | | `/txs/summary` | dialect | **Not** in Blockstream Esplora [`API.md`](https://github.com/Blockstream/esplora/blob/master/API.md). Compact `{txid, value, height, time}` like mempool.space `/address/:addr/txs/summary`. Confirmed only (25/page, newest first); path cursor `/:last_seen_txid` and mempool.space `?after_txid=` (unknown → **422**). `value` is net sats for that script in that tx (funded − spent). `time` is the confirming header timestamp (`0` if the header is missing). Mempool rows stay on `/txs` and `/txs/mempool`. Over `--max-sh-creates` → **503**. | | Mempool / fees | done | `/mempool`, `/mempool/txids`, `/mempool/txids/page[/:last]`, `/mempool/recent` (accept-order ring), `/fee-estimates`, mempool.space `/fees/recommended` and `/v1/fees/recommended` (sat/vB tiers) | diff --git a/OPERATOR.md b/OPERATOR.md index d68e1790e..dc5b95b14 100644 --- a/OPERATOR.md +++ b/OPERATOR.md @@ -695,7 +695,7 @@ counts, ingest OA, and refuse lines: [`SCHEMA.md`](./SCHEMA.md) and ## Schema upgrade -Live bytes: [`SCHEMA.md`](./SCHEMA.md) (`SCHEMA_VERSION = 24`). This section is +Live bytes: [`SCHEMA.md`](./SCHEMA.md) (`SCHEMA_VERSION = 25`). This section is the operator copy-paste only — do not treat it as a second layout map. Open **never silently wipes** a populated store (policy: @@ -705,23 +705,24 @@ names the dirs. Corrupt files are **not** repaired in-process. | Incoming `meta` | What this binary does | |-----------------|------------------------| -| **24** | Open. | -| **23** | Rewrite `meta` to 24 first, then rewrite `header.body` 88 B rows to 96 B (size/weight 0) on open. Class A tx stems kept. A torn `header.body` rewrite is retried. | -| **22**, occupied Class A | Rewrite `meta` to 24 first, then `create.loc.ovf` 12 B→16 B on `TxTable::open`, then `header.body` 88→96. Crash window is 24 `meta` + old ovf/header; this binary retries those file rewrites. | -| **22**, empty Class A | Rewrite `meta` to 24, then open. | -| **21**, empty Class A | Unlink leftover `spent.off`, rewrite `meta` to 24, then open. | +| **25** | Open. | +| **24** | Rewrite `meta` to 25, then create/extend zeroed `txstat.body` to `create.loc` count (no `txout.body` rewrite). Unlink leftover `txfixed.body`. | +| **23** | Rewrite `meta` to 25 first, then rewrite `header.body` 88 B rows to 96 B (size/weight 0) on open. Class A tx stems kept. A torn `header.body` rewrite is retried. Zero-extend `txstat.body`. | +| **22**, occupied Class A | Rewrite `meta` to 25 first, then `create.loc.ovf` 12 B→16 B on `TxTable::open`, then `header.body` 88→96. Crash window is 25 `meta` + old ovf/header; this binary retries those file rewrites. Zero-extend `txstat.body`. | +| **22**, empty Class A | Rewrite `meta` to 25, then open. | +| **21**, empty Class A | Unlink leftover `spent.off`, rewrite `meta` to 25, then open. | | **21**, occupied Class A | **Refuse.** Wipe datadir and redo IBD. | -| **20**, empty Class A | Unlink leftover `spent.off`, rewrite `meta` to 24, then open. | +| **20**, empty Class A | Unlink leftover `spent.off`, rewrite `meta` to 25, then open. | | **20**, occupied Class A | **Refuse.** Wipe datadir and redo IBD. | -| **19** or **18**, empty Class A and empty `tx.head` / no `scripthash*` data | Rewrite `meta` to 24, then open. | +| **19** or **18**, empty Class A and empty `tx.head` / no `scripthash*` data | Rewrite `meta` to 25, then open. | | **19** or **18**, occupied Class A | **Refuse.** Wipe datadir and redo IBD. | | **19** or **18**, empty Class A, occupied `tx.head` or any `scripthash*` | **Refuse.** Wipe `store/tx.head` and `store/scripthash*`, keep Class A, restart. | -| **17**, empty Class A and empty `tx.head` / no `scripthash*` data | Rewrite `meta` to 24, then open. | +| **17**, empty Class A and empty `tx.head` / no `scripthash*` data | Rewrite `meta` to 25, then open. | | **17**, occupied Class A | **Refuse.** Wipe datadir and redo IBD. | | **17**, empty Class A, populated `tx.head` or any `scripthash*` | **Refuse.** Wipe those index dirs, keep Class A, restart. | | Older than 17 with creates / leftover catalogs | **Refuse.** The error names files; often a full datadir wipe + IBD. Details: SCHEMA.md **13/14→17**, **15→17**, **16→17**. | -A **23 binary** refuses 24 `meta` (do not downgrade in place). A **22 binary** refuses 23+ `meta`. A **21 binary** refuses 22+ `meta`. A **19 binary** refuses 20+ `meta`. +A **24 binary** refuses 25 `meta` (do not downgrade in place). A **23 binary** refuses 24+ `meta`. A **22 binary** refuses 23+ `meta`. A **21 binary** refuses 22+ `meta`. A **19 binary** refuses 20+ `meta`. When the schema-22 Class A refuse fires, the log line is: diff --git a/SCHEMA.md b/SCHEMA.md index 18156c9ff..80168a2e3 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -1,7 +1,12 @@ # On-disk schema (current) -**Version:** `SCHEMA_VERSION = 24` (`rbitcoin_primitives`). -**Status:** 24 is `header.body` 96 B (trailing `size:u32` + `weight:u32`). Occupied +**Version:** `SCHEMA_VERSION = 25` (`rbitcoin_primitives`). +**Status:** 25 is `txstat.body` 8 B/create (canonical ULEB `n_in`/`fee_sat`/`base`/`wit_extra`; +per-header remaining-byte overflow in `txstat.ovf` + `txstat.blk`). Occupied +24 rewrites `meta` and zero-extends `txstat.body` to `create.loc` count (**no** +`txout.body` rewrite; leftover LAYOUT17 still has uleb `input_count`). Unreleased +leftover `txfixed.body` is unlinked. A 24 +binary refuses 25 `meta`. 24 is `header.body` 96 B (trailing `size:u32` + `weight:u32`). Occupied 23 rewrites 88 B rows via `header.body.grow` then rename (size/weight 0 until confirm stamps or lazy fill). SH extent last-page reserved (offset 20) is create count (`0` = unknown; readers walk, appender stamps on pack/append). 23 is `create.loc.ovf` 16 B (`fk:u64` + strides/`n_out` @@ -13,13 +18,13 @@ slots flags + u40 spend fk + u16 vin (still 8 bytes). `txout` amount is flags bi is canonical compact: strip trailing tens up to `e=9` (`e<9` and mantissa divisible by 10 is Corrupt; zero is `e=0`, mantissa 0). Occupied 15–21 LAYOUT17 Class A with creates is **refused** -(wipe datadir and redo IBD). Empty 15–23 rewrite `meta` to 24 and unlink leftover +(wipe datadir and redo IBD). Empty 15–24 rewrite `meta` to 25 and unlink leftover `spent.off` and leftover `*.idx`. A 23 binary refuses 24 `meta`. Occupied schema 18/19 `tx.head` or `scripthash*` (empty Class A) is **refused** (wipe those index -dirs, keep Class A). Empty 18/19 indexes rewrite `meta` to 24; `tx.head` rebuilds +dirs, keep Class A). Empty 18/19 indexes rewrite `meta` to 25; `tx.head` rebuilds from Class A; SH rematerializes with `--sh-index`. An 19 binary refuses 20+ `meta`. A 17 datadir with populated `tx.head` or `scripthash*` and empty -Class A is **refused**. Empty 17 indexes rewrite `meta` to 24. +Class A is **refused**. Empty 17 indexes rewrite `meta` to 25. Operator copy-paste (which dirs to wipe; kill-9 is not a migrate): [`OPERATOR.md`](./OPERATOR.md#schema-upgrade). @@ -73,11 +78,12 @@ index refuses pack8 Paged (mode 10) scripthash heads; wipe store/scripthash* the ``` **21→22 open:** occupied Class A with creates: `schema 22 refuses schema-21 Class A with creates; wipe datadir and redo IBD`. -Empty 21 rewrites `store/meta` to 24 and unlinks leftover `spent.off`. -Table file headers 13–24 remain `schema_file_openable`. A 22 binary refuses 23 `meta`. -Occupied 15–20 LAYOUT17 Class A with creates hits the same refuse (old flags+u56-fk / no vin pack). Empty 15–20 rewrite `meta` to 24. +Empty 21 rewrites `store/meta` to 25 and unlinks leftover `spent.off`. +Table file headers 13–25 remain `schema_file_openable`. A 22 binary refuses 23 `meta`. +Occupied 15–20 LAYOUT17 Class A with creates hits the same refuse (old flags+u56-fk / no vin pack). Empty 15–20 rewrite `meta` to 25. **22→23 open:** occupied Class A rewrites `create.loc.ovf` 12 B rows (`fk:u64` + two u16) to 16 B (`fk:u64` + two u32) and `store/meta` to 23. Empty 22 rewrites `meta`. A 22 binary refuses 23 `meta`. Spent vin stays u16 (stripped input ≥ ~41 B ⇒ ≲24k vins in a 1 MB block; widening would bump the 8 B spent slot). -**23→24 open:** rewrite `header.body` 88 B rows to 96 B (`size:u32` + `weight:u32` = 0) via `header.body.grow` then rename; rewrite `meta` to 24. Class A tx stems kept. Empty 23 rewrites `meta`. A 23 binary refuses 24 `meta`. Crash with leftover `.grow` discards it and retries; 96-byte body with meta 23 only rewrites `meta`. +**23→24 open:** rewrite `header.body` 88 B rows to 96 B (`size:u32` + `weight:u32` = 0) via `header.body.grow` then rename; rewrite `meta` to 25. Class A tx stems kept. Empty 23 rewrites `meta`. A 23 binary refuses 24 `meta`. Crash with leftover `.grow` discards it and retries; 96-byte body with meta 23 only rewrites `meta`. +**24→25 open:** rewrite `meta` to 25; create or zero-extend `txstat.body` to `create.loc` count. Do **not** rewrite `txout.body`. Unlink leftover `txfixed.body`. A 24 binary refuses 25 `meta`. **Endianness:** little-endian for all multi-byte integers. Older versions and migration notes live in [`SCHEMA_HISTORY.md`](./SCHEMA_HISTORY.md). @@ -202,6 +208,7 @@ itself changed. spent.body # sole-spender 8 B × n_out; leftover spent.off unlinked tx.body / tx.idx.* # schema ≤14 packed (refused if non-empty) txid.body # dense create_fk-ordered txids (schema 13+) + txstat.body / txstat.ovf / txstat.blk # 8 B/create ULEB econ + per-header tails (schema 25) tx.head/ # meta + open OA NNNNNN; sealed NNNNNN.mphf|.fuse8 spent.ovf # multi-spender overflow (was spenders.body) confirmed.body # Class C: height → header_fk @@ -267,6 +274,9 @@ the hot volume. | 16 | inwit (`inwit.body`) | | 17 | spent (`spent.body`) | | 18 | delta loc (`create.loc` / `inwit.loc` and `.ovf`) | +| 19 | txstat (`txstat.body`, 8 B/create) | +| 20 | txstat overflow (`txstat.ovf`) | +| 21 | txstat per-header locator (`txstat.blk`, 16 B/header) | --- @@ -331,7 +341,31 @@ offset 0..32 — 32-byte file header (standard 16-byte TableFile header + 16 offset 32+(fk-1)*32 — txid for create_fk = fk (1-based) ``` -Append-published with Class A body/idx on the sole Class A write path. Count must match `txout` / `inwit` / `spent` / `txid.body`. Head-resolve multi-cand identity peeks this file (fixed offset), **not** a body prefix. +Append-published with Class A body/idx on the sole Class A write path. Count must match `txout` / `inwit` / `spent` / `txid.body` / `txstat.body`. Head-resolve multi-cand identity peeks this file (fixed offset), **not** a body prefix. + +### Confirm-time econ (`txstat.body`, schema 25) + +```text +txstat.body offset 0..32 — TableFile 16 + 16 pad + offset 32+(fk-1)×8 — 8-byte cell +txstat.ovf append-only — remaining ULEB bytes when the stream exceeds 8 B +txstat.blk offset 32+(header_fk-1)×16 — off:u64, len:u32, n_ovf:u32 +``` + +Cell payload is four canonical ULEBs: `n_in`, `fee_sat`, `base` (non-witness +size), `wit_extra` (`total_size − base`). Readers derive `size = base + wit_extra` +and `weight = 4×base + wit_extra`. `n_in` is first and ≤ 3 B (`≤ u16::MAX`, same +cap as `spent.vin`) so pin / SH / tweaks parse it from the cell without opening +`.ovf`. All-zero cell = unstamped. A truncated ULEB or fewer than four fields +means the rest of the stream is in that header's overflow blob (`encoded[8..]` +only). Missing tail is `Corrupt("invariant: txstat overflow missing")`. Overlong +ULEB is Corrupt. Trailing non-zero after four fields is Corrupt. Placeholders +(`n_in ≠ 0`, rest zero) always fit. Class A append of placeholders never overflows; +only a confirm stamp can emit tails. Blob entries are `u16 index_in_block` + +`u8 nrest` + rest, in block-index order. Empty header: `len=0`. Occupied 24 open +extends zeros to loc count (~11.3 GiB at the 2026-08-13 census if fully allocated). +Pin / SH / tweaks do **not** open these files. Unreleased leftover `txfixed.body` +is unlinked on open. ### Split bodies (schema 15) @@ -366,11 +400,15 @@ Decode walks meta + runs to a logical end; any remaining bytes in the loc span m **Body meta (schema 22 LAYOUT17, variable):** first byte bit 7 = `LAYOUT17` (required). Bits 0–2 encode version 1/2/3 (else explicit i32 LE); bit 3 = -locktime 0 (else uleb locktime); then uleb `input_count` only (no -`output_count`). Typical v2+locktime 0 is **2 B**. `CreateLocPair.n_out` (≥ 1) -fills `TxRecord.output_count`. Schema-15 16-byte prefixes (v1 starts -`01 00 00 00`) are not accepted. `input_start_fk` / `output_start_fk` stay null -in RAM. Soft `TxRecord.txid` is filled from the sidefile on get paths. +locktime 0 (else uleb locktime). Bit 4 (`N_IN_TXSTAT`) omits the following +uleb `input_count` (`n_in` is on `txstat.body`; decode reports 0 until a +reader fills from the stamped row). Bits 5–6 reserved (nonzero → Corrupt). +New 25 writes omit the uleb (v2+locktime 0 is **1 B**). Leftover 24 rows +keep the uleb (typical v2+locktime 0 is **2 B**). +`CreateLocPair.n_out` (≥ 1) fills `TxRecord.output_count`. Schema-15 16-byte +prefixes (v1 starts `01 00 00 00`) are not accepted. `input_start_fk` / +`output_start_fk` stay null in RAM. Soft `TxRecord.txid` is filled from the +sidefile on get paths. ### Create / inwit locators (`create.loc` / `inwit.loc`) diff --git a/SCHEMA_HISTORY.md b/SCHEMA_HISTORY.md index 5d87a5b5b..80a3ebf5b 100644 --- a/SCHEMA_HISTORY.md +++ b/SCHEMA_HISTORY.md @@ -1,7 +1,7 @@ # Schema history Historic on-disk layouts for the rbitcoin chain store. -**Current layout:** [`SCHEMA.md`](./SCHEMA.md) (`SCHEMA_VERSION = 24`). +**Current layout:** [`SCHEMA.md`](./SCHEMA.md) (`SCHEMA_VERSION = 25`). Until 1.0 there is **no in-place migration**: a new major layout generally means wipe the store and redo IBD. This file is for archaeology, code archaeology, and understanding why the current design looks the way it does. @@ -13,7 +13,8 @@ Versions below are listed **newest → oldest** after the summary table. | Version | Headline change | Still in current tree as… | |--------:|-----------------|---------------------------| -| **24** | `header.body` 96 B (`size`/`weight` u32). Occupied 23 rewrites 88 B rows. Extent last-page reserved = create count. | **Current** | +| **25** | `txstat.body` 8 B/create ULEB + per-header overflow. Occupied 24 zero-extends; no `txout` rewrite. | **Current** | +| **24** | `header.body` 96 B (`size`/`weight` u32). Occupied 23 rewrites 88 B rows. Extent last-page reserved = create count. | Prior | | **23** | `create.loc.ovf` 16 B (u32 strides / `n_out`). Occupied 22 rewrites 12 B ovf. Occupied 15–21 Class A refused. | Prior | | **22** | `create.loc` + `inwit.loc`; no Class A `*.idx`. LAYOUT17 drops `output_count`. Spent slot flags + u40 fk + u16 vin. `txout` amount is exp nibble + ULEB mantissa. Occupied 15–21 Class A refused. Empty 15–21 rewrite `meta` and unlink leftover `spent.off` + leftover `*.idx`. | Prior | | **21** | Drop `spent.idx`. Spent ranges are `8 × max(n_out,1)` from txout meta; sparse `spent.off`. Unlink leftover idx; rewrite `meta` 20→21. Table headers 13–20 remain openable. | Prior | @@ -38,6 +39,17 @@ Versions below are listed **newest → oldest** after the summary table. --- +## v25 (`txstat.body` 8 B/create) + +Dense `create_fk`-addressed econ sidefile, same header pad as `txid.body` +(32 B prefix, 8 B ULEB cells). Stream is four canonical ULEBs: `n_in`, +`fee_sat`, `base`, `wit_extra`. Overflow remaining bytes live in a per-header +blob (`txstat.ovf` + `txstat.blk`). Occupied 24 rewrites `meta` and extends +zeros to `create.loc` count. `txout.body` is not rewritten. Unreleased leftover +`txfixed.body` is unlinked. A 24 binary refuses 25 `meta`. Named disk: ~11.3 GiB +at the 2026-08-13 census. Pin/SH/tweaks do not open this file. New writes set +LAYOUT17 bit 4 and omit the `input_count` uleb; leftover 24 rows still decode. + ## v24 (header size/weight + SH extent create count) `header.body` records are 96 B: schema-23 consensus fields (88 B) plus trailing diff --git a/crates/rbitcoin-consensus/src/confirm_run/pin.rs b/crates/rbitcoin-consensus/src/confirm_run/pin.rs index ec7c675e5..0e3ef3ae8 100644 --- a/crates/rbitcoin-consensus/src/confirm_run/pin.rs +++ b/crates/rbitcoin-consensus/src/confirm_run/pin.rs @@ -173,7 +173,7 @@ fn apply_plan_pins( if !need.is_empty() && batch_parents.pin_covered(fk, need) { if let Some(pin) = plan_by_id.get(id) { let tx = pin.tx(); - let cb = if tx.input_count != 1 { + let cb = if tx.input_count > 1 { Some(false) } else { None @@ -191,7 +191,7 @@ fn apply_plan_pins( still_need.insert(*id, need.clone()); continue; } - let cb = if pin.tx().input_count != 1 { + let cb = if pin.tx().input_count > 1 { Some(false) } else { None @@ -288,7 +288,7 @@ fn denserels_by_stamped_range( "invariant: lookup stage miss (load parent create identity not stamped)", )))?; } - let cb = if tx.input_count != 1 { + let cb = if tx.input_count > 1 { Some(false) } else { None diff --git a/crates/rbitcoin-consensus/src/confirm_run/write.rs b/crates/rbitcoin-consensus/src/confirm_run/write.rs index 7579a6e21..ebafbe259 100644 --- a/crates/rbitcoin-consensus/src/confirm_run/write.rs +++ b/crates/rbitcoin-consensus/src/confirm_run/write.rs @@ -95,7 +95,7 @@ fn apply_archive_plan( ns.plan_take_ns = t_take.elapsed().as_nanos() as u64; let t_ca = Instant::now(); let (committed, loc) = query - .archive_commit_plan_defer_head(plan) + .archive_commit_plan_defer_head_parents(plan, Some(&batch.batch_parents)) .map_err(ConsensusError::from)?; ns.class_a_ns = t_ca.elapsed().as_nanos() as u64; if !committed { @@ -478,7 +478,7 @@ pub(super) fn fill_planned_create_layout_after_commit( let mut checked = vouts.clone(); checked.sort_unstable(); checked.dedup(); - let cb = if pin.tx().input_count != 1 { + let cb = if pin.tx().input_count > 1 { Some(false) } else { None diff --git a/crates/rbitcoin-consensus/src/confirm_run/write_idempotent_tests.rs b/crates/rbitcoin-consensus/src/confirm_run/write_idempotent_tests.rs index 7abd1c392..e96df86f7 100644 --- a/crates/rbitcoin-consensus/src/confirm_run/write_idempotent_tests.rs +++ b/crates/rbitcoin-consensus/src/confirm_run/write_idempotent_tests.rs @@ -3186,3 +3186,34 @@ fn already_at_height_retries_post_commit_spend_annotate() { assert_eq!(field2, spend_fk, "post_commit must annotate after Class C"); let _ = std::fs::remove_dir_all(&path); } + +#[test] +fn confirm_write_stamps_txstat_fee_size() { + use crate::{ + accept_and_connect_block, confirm_wire_run, genesis_block, mine_empty_regtest, ChainParams, + Milestone, + }; + use rbitcoin_primitives::{Fk, Height}; + + let (path, q) = tmp_query(); + let params = ChainParams::regtest(); + let ms = Milestone::NONE; + let genesis = genesis_block(¶ms); + accept_and_connect_block(&q, ¶ms, Height::GENESIS, &genesis, ms).unwrap(); + let g = q.get_txstat(Fk(1)).unwrap().expect("genesis txstat"); + let gtx = &genesis.txdata[0]; + assert_eq!(g.fee_sat, 0); + assert_eq!(g.n_in, 1); + assert_eq!(g.size() as usize, gtx.total_size()); + assert_eq!(g.weight(), gtx.weight().to_wu()); + + let b1 = mine_empty_regtest(genesis.block_hash(), genesis.header.time + 600, 1); + confirm_wire_run(&q, ¶ms, ms, &[(Height(1), b1.clone())]).unwrap(); + let row = q.get_txstat(Fk(2)).unwrap().expect("height-1 txstat"); + let tx = &b1.txdata[0]; + assert_eq!(row.fee_sat, 0); + assert_eq!(row.n_in, 1); + assert_eq!(row.size() as usize, tx.total_size()); + assert_eq!(row.weight(), tx.weight().to_wu()); + let _ = std::fs::remove_dir_all(&path); +} diff --git a/crates/rbitcoin-esplora/src/server.rs b/crates/rbitcoin-esplora/src/server.rs index 1ae22e93a..541757bc3 100644 --- a/crates/rbitcoin-esplora/src/server.rs +++ b/crates/rbitcoin-esplora/src/server.rs @@ -3843,7 +3843,7 @@ mod tests { let row: serde_json::Value = serde_json::from_str(&body).unwrap(); assert_eq!(row["txid"], tx0); assert_eq!(row["pruned"], true); - assert!(row.get("vin").is_none()); + assert_eq!(row["vin"], serde_json::json!([])); assert!(row.get("vout").is_some()); let (st, body) = http_get(addr, &format!("/tx/{tx0}/raw")).await; diff --git a/crates/rbitcoin-esplora/src/tx_json.rs b/crates/rbitcoin-esplora/src/tx_json.rs index 547035cde..db3c43c16 100644 --- a/crates/rbitcoin-esplora/src/tx_json.rs +++ b/crates/rbitcoin-esplora/src/tx_json.rs @@ -181,14 +181,23 @@ fn build_tx_json_pruned(query: &Query, tx_fk: Fk, network: Network) -> Result out.push_str(&format!(" {name}={v}ms(txstat={})", s.arch_write_txstat_ms)), "ensure" => out.push_str(&format!( " {name}={v}ms(pin={} cold={})", s.ensure_res_hit, s.ensure_cold_n @@ -1186,6 +1190,10 @@ fn append_write_inventory_debug(out: &mut String, s: &IbdPerfSample, us: impl Fn for (name, _, ns) in WriteStageSample::INVENTORY { let v = us(ns(&s.write)); match *name { + "class_a" => out.push_str(&format!( + " {name}={v}(txstat={})", + us(s.arch_write_txstat_ms.saturating_mul(1_000_000)) + )), "struct" => out.push_str(&format!( " {name}={v} spent={} create_h={} bip68={}", us(s.structural_spent_ns), @@ -1457,10 +1465,11 @@ pub(crate) fn format_info(s: &IbdPerfSample) -> String { append_nz(&mut out, "ovl_n", s.ovl_n); if s.arch_write_body_ms > 0 || s.arch_write_head_ms > 0 || s.arch_write_htxs_ms > 0 { out.push_str(&format!( - " class_a_sub(body={} head={} htxs={} reserve={})", + " class_a_sub(body={} head={} htxs={} txstat={} reserve={})", s.arch_write_body_ms, s.arch_write_head_ms, s.arch_write_htxs_ms, + s.arch_write_txstat_ms, s.arch_write_reserve_ms, )); } @@ -1637,12 +1646,13 @@ pub(crate) fn format_debug(s: &IbdPerfSample) -> String { let ca_head_us_blk = div_or_0(s.arch_write_head_ms * 1000, s.arch_write_blocks); let ca_body_us_blk = div_or_0(s.arch_write_body_ms * 1000, s.arch_write_blocks); out.push_str(&format!( - " | class_a_commit total={} body={} head={} htxs={} reserve={} spend={} flush={} blks={} \ + " | class_a_commit total={} body={} head={} htxs={} txstat={} reserve={} spend={} flush={} blks={} \ ca_head_us/blk={} ca_body_us/blk={}", s.arch_write_total_ms, s.arch_write_body_ms, s.arch_write_head_ms, s.arch_write_htxs_ms, + s.arch_write_txstat_ms, s.arch_write_reserve_ms, s.arch_write_spend_ms, s.arch_write_flush_ms, diff --git a/crates/rbitcoin-primitives/src/lib.rs b/crates/rbitcoin-primitives/src/lib.rs index ea0b6a774..27b42ecfb 100644 --- a/crates/rbitcoin-primitives/src/lib.rs +++ b/crates/rbitcoin-primitives/src/lib.rs @@ -78,6 +78,11 @@ pub const STORE_MAGIC: [u8; 4] = *b"RBT1"; /// Current on-disk schema version. Live layout: workspace `SCHEMA.md`. /// Historic versions: `SCHEMA_HISTORY.md`. /// +/// **25:** `txstat.body` 8 B/create (ULEB `n_in`/`fee_sat`/`base`/`wit_extra`, +/// per-header remaining-byte overflow). Occupied 24 rewrites `meta` +/// and extends a zeroed `txstat.body` to loc count (no `txout.body` +/// rewrite). Unreleased leftover `txfixed.body` is unlinked. A 24 +/// binary refuses 25 `meta`. /// **24:** `header.body` 96 B (trailing `size:u32` + `weight:u32`). Occupied 23 /// rewrites 88 B rows (`header.body.grow` then rename); zeros until /// confirm stamps. SH extent last-page reserved is create count. @@ -107,11 +112,12 @@ pub const STORE_MAGIC: [u8; 4] = *b"RBT1"; /// Refuse packed schema-13/14 Class A with txs; refuse materialized page-era SH. /// **14:** Class B SH head = Empty/Inline/Paged (4 KiB page chains); refuse schema-13 slabs. /// **13:** dense `txid.body` sidefile; Class A packed body meta **without** leading txid. -pub const SCHEMA_VERSION: u16 = 24; +pub const SCHEMA_VERSION: u16 = 25; /// True if `ver` may appear in store `meta` / table headers this binary can open. /// -/// Schema **24** is current (`header.body` 96 B). Occupied **23** rewrites +/// Schema **25** is current (`txstat.body` 8 B/create). Occupied **24** +/// rewrites `meta` and zero-extends `txstat.body`. Occupied **23** rewrites /// 88 B header rows. Occupied **22** Class A /// rewrites 12 B ovf rows. Occupied 21 Class A is refused. Schema **21** empty /// Class A rewrites `meta`. Schema **20** table headers still open when Class A @@ -279,6 +285,12 @@ pub enum TableKind { Spent = 17, /// Create/inwit delta locators (`create.loc` / `inwit.loc` and `.ovf`). DeltaLoc = 18, + /// Dense create_fk-ordered confirm-time econ (`txstat.body`, 8 B). + TxStat = 19, + /// Per-header remaining-byte overflow (`txstat.ovf`). + TxStatOvf = 20, + /// Per-header overflow locator (`txstat.blk`, 16 B/header). + TxStatBlk = 21, } impl TableKind { @@ -300,6 +312,9 @@ impl TableKind { 16 => Some(TableKind::Inwit), 17 => Some(TableKind::Spent), 18 => Some(TableKind::DeltaLoc), + 19 => Some(TableKind::TxStat), + 20 => Some(TableKind::TxStatOvf), + 21 => Some(TableKind::TxStatBlk), _ => None, } } @@ -355,7 +370,9 @@ mod tests { #[test] fn table_kind_roundtrip() { - for v in [1u16, 2, 3, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18] { + for v in [ + 1u16, 2, 3, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, + ] { let k = TableKind::from_u16(v).expect("kind"); assert_eq!(k.as_u16(), v); } @@ -372,13 +389,17 @@ mod tests { assert_eq!(TableKind::Inwit.as_u16(), 16); assert_eq!(TableKind::Spent.as_u16(), 17); assert_eq!(TableKind::DeltaLoc.as_u16(), 18); + assert_eq!(TableKind::TxStat.as_u16(), 19); + assert_eq!(TableKind::TxStatOvf.as_u16(), 20); + assert_eq!(TableKind::TxStatBlk.as_u16(), 21); } #[test] fn constants_stable() { assert_eq!(STORE_MAGIC, *b"RBT1"); - assert_eq!(SCHEMA_VERSION, 24); + assert_eq!(SCHEMA_VERSION, 25); assert!(!VERSION.is_empty()); + assert!(schema_file_openable(25)); assert!(schema_file_openable(24)); assert!(schema_file_openable(23)); assert!(schema_file_openable(22)); @@ -392,10 +413,22 @@ mod tests { assert!(schema_file_openable(14)); assert!(schema_file_openable(13)); assert!(!schema_file_openable(12)); - assert!(!schema_file_openable(25)); + assert!(!schema_file_openable(26)); assert!(!schema_file_openable(0)); } + #[test] + fn open_schema25_meta_refused_by_v24_gate() { + const SCHEMA24_MAX: u16 = 24; + fn schema24_binary_openable(ver: u16) -> bool { + (13..=SCHEMA24_MAX).contains(&ver) + } + assert!(!schema24_binary_openable(25)); + assert!(schema24_binary_openable(24)); + assert!(schema_file_openable(24)); + assert!(schema_file_openable(25)); + } + #[test] fn subversion_comments_and_rejects() { assert_eq!( diff --git a/crates/rbitcoin-query/src/archive.rs b/crates/rbitcoin-query/src/archive.rs index 26ac20cf8..81ce6e138 100644 --- a/crates/rbitcoin-query/src/archive.rs +++ b/crates/rbitcoin-query/src/archive.rs @@ -154,7 +154,7 @@ impl CreatePinInner { } } - fn wire_tx(&self) -> Option<&bitcoin::Transaction> { + pub(crate) fn wire_tx(&self) -> Option<&bitcoin::Transaction> { match self { Self::Wire { block, tx_index, .. @@ -660,6 +660,131 @@ fn tx_record_from_wire(tx: &bitcoin::Transaction, txid: [u8; 32]) -> TxRecord { } } +fn stamp_txstat_rows( + query: &Query, + packed: &[(CreatePin, Vec)], + planned: &[Fk], + parents: Option<&crate::BatchParents>, +) -> Result, QueryError> { + let mut idx: crate::U64Map = crate::U64Map::default(); + for (i, fk) in planned.iter().enumerate() { + if let Some(id) = fk.get() { + idx.insert(id, i); + } + } + let mut out = Vec::with_capacity(packed.len()); + for (pin, ins) in packed { + out.push(stamp_one_txstat(query, packed, &idx, parents, pin, ins)?); + } + Ok(out) +} + +fn stamp_one_txstat( + query: &Query, + packed: &[(CreatePin, Vec)], + idx: &crate::U64Map, + parents: Option<&crate::BatchParents>, + pin: &CreatePin, + ins: &[InputRecord], +) -> Result { + let Some(tx) = pin.wire_tx() else { + return Ok(txstat_placeholder_query(pin.tx().input_count)); + }; + let in_sum = if tx.is_coinbase() { + None + } else { + let mut in_sum = 0u64; + for inp in ins { + if inp.is_coinbase() { + return Err(StoreError::Corrupt("invariant: mixed coinbase vin")); + } + let val = prevout_value(query, packed, idx, parents, inp)?; + in_sum = in_sum + .checked_add(val) + .ok_or(StoreError::Corrupt("txstat in_sum overflow"))?; + } + Some(in_sum) + }; + txstat_row_from_tx(tx, in_sum) +} + +pub(crate) fn txstat_row_from_tx( + tx: &bitcoin::Transaction, + in_sum: Option, +) -> Result { + let n_in = tx.input.len() as u32; + if n_in > u32::from(u16::MAX) { + return Err(StoreError::Corrupt("txstat n_in exceeds 16 bits")); + } + let base_sz = tx.base_size(); + let total = tx.total_size(); + let base = u32::try_from(base_sz).map_err(|_| StoreError::Corrupt("txstat base"))?; + let wit_extra = u32::try_from(total.saturating_sub(base_sz)) + .map_err(|_| StoreError::Corrupt("txstat wit_extra"))?; + let out_sum: u64 = tx.output.iter().map(|o| o.value.to_sat()).sum(); + let fee = match in_sum { + None => 0u64, + Some(in_sum) => { + if in_sum < out_sum { + return Err(StoreError::Corrupt("txstat fee underflow")); + } + in_sum - out_sum + } + }; + Ok(rbitcoin_store::TxStatRow { + n_in, + fee_sat: fee, + base, + wit_extra, + }) +} + +fn txstat_placeholder_query(n_in: u32) -> rbitcoin_store::TxStatRow { + rbitcoin_store::TxStatRow { + n_in, + fee_sat: 0, + base: 0, + wit_extra: 0, + } +} + +fn prevout_value( + query: &Query, + packed: &[(CreatePin, Vec)], + idx: &crate::U64Map, + parents: Option<&crate::BatchParents>, + inp: &InputRecord, +) -> Result { + let Some(cid) = inp.create_fk.get() else { + return Err(StoreError::Corrupt("invariant: spend missing create_fk")); + }; + if let Some(&i) = idx.get(&cid) { + let (val, _) = packed[i] + .0 + .out_parts(inp.prev_index) + .ok_or(StoreError::Corrupt("invariant: same-batch prevout"))?; + if val < 0 { + return Err(StoreError::Corrupt("txstat prevout negative")); + } + return Ok(val as u64); + } + if let Some(p) = parents { + if let Some(hit) = + p.get_parent_txout_parts(inp.create_fk, inp.prev_index, |v, s, _| (v, s.to_vec())) + { + if hit.0 < 0 { + return Err(StoreError::Corrupt("txstat prevout negative")); + } + return Ok(hit.0 as u64); + } + } + let o = query.tx_output_at_fk(inp.create_fk, inp.prev_index)?; + if o.value < 0 { + return Err(StoreError::Corrupt("txstat prevout negative")); + } + Ok(o.value as u64) +} + impl Query { /// Class A plan + commit from wire blocks. Does not set tip. pub fn archive_class_a_from_wire(&self, items: &[WirePlanNeed<'_>]) -> Result<(), QueryError> { @@ -974,8 +1099,17 @@ impl Query { /// Like [`Self::archive_commit_plan`] but leaves `tx.head` in the pending map. /// Loc pairs are the Class A append starts (RAM). Empty when nothing committed. pub fn archive_commit_plan_defer_head( + &self, + plan: ArchiveWritePlan, + ) -> Result<(bool, Vec), QueryError> { + self.archive_commit_plan_defer_head_parents(plan, None) + } + + /// Class A commit with optional parent pins for `txstat` fee. + pub fn archive_commit_plan_defer_head_parents( &self, mut plan: ArchiveWritePlan, + parents: Option<&crate::BatchParents>, ) -> Result<(bool, Vec), QueryError> { use std::time::Instant; if plan.packed.is_empty() { @@ -998,10 +1132,17 @@ impl Query { let t = Instant::now(); let overlay = plan.same_batch_spent_overlay(); - let (got_tx_fks, loc) = self.store.put_tx_full_batch_from_pins( + let txstat = stamp_txstat_rows(self, &plan.packed, &plan.planned_fks, parents)?; + let txstat_ns = t.elapsed().as_nanos() as u64; + self.confirm_stats().note_write_txstat(txstat_ns); + + let t = Instant::now(); + let (got_tx_fks, loc) = self.store.put_tx_full_batch_from_pins_with_txstat( &plan.packed, /*index=*/ false, &overlay, + &txstat, + &plan.per_header_ranges, )?; let body_ns = t.elapsed().as_nanos() as u64; if got_tx_fks.len() != plan.packed.len() { diff --git a/crates/rbitcoin-query/src/combined_stage.rs b/crates/rbitcoin-query/src/combined_stage.rs index 3644e44bc..cd7db7367 100644 --- a/crates/rbitcoin-query/src/combined_stage.rs +++ b/crates/rbitcoin-query/src/combined_stage.rs @@ -104,7 +104,7 @@ pub fn load_creates_once( let mut decoded_outs = None; match mode { IdxBodyMode::Full => { - if let Ok((tx, _empty_ins, outs, rels)) = + if let Ok((mut tx, _empty_ins, outs, rels)) = decode_packed_tx_with_spender_rels_secret(&job.body, job.n_out, Some(secret)) { let Some(ij) = inwit_jobs.get(i) else { @@ -122,6 +122,9 @@ pub fn load_creates_once( .map_err(|_| { StoreError::Corrupt("invariant: packed create inwit decode failed") })?; + if tx.input_count == 0 { + tx.input_count = ins.len() as u32; + } decoded_full = Some((tx, ins, outs, rels)); } else { return Err(StoreError::Corrupt( diff --git a/crates/rbitcoin-query/src/confirm_stats.rs b/crates/rbitcoin-query/src/confirm_stats.rs index 3d337d7a9..088b4c127 100644 --- a/crates/rbitcoin-query/src/confirm_stats.rs +++ b/crates/rbitcoin-query/src/confirm_stats.rs @@ -248,6 +248,7 @@ confirm_window! { arch_write_head_ns, arch_write_spend_ns, arch_write_htxs_ns, + arch_write_txstat_ns, arch_write_flush_ns, arch_write_blocks, // lookup wave @@ -520,6 +521,10 @@ impl ConfirmStats { add(&self.arch_write_blocks, blocks); } + pub fn note_write_txstat(&self, ns: u64) { + add(&self.arch_write_txstat_ns, ns); + } + pub fn note_write_flush(&self, ns: u64) { add(&self.arch_write_flush_ns, ns); add(&self.arch_write_total_ns, ns); diff --git a/crates/rbitcoin-query/src/lib.rs b/crates/rbitcoin-query/src/lib.rs index bece46efc..48b5b2b46 100644 --- a/crates/rbitcoin-query/src/lib.rs +++ b/crates/rbitcoin-query/src/lib.rs @@ -27,6 +27,7 @@ mod write_create_loc; #[cfg(debug_assertions)] pub use combined_stage::{body_ok_reads, reset_body_ok_reads}; pub use combined_stage::{load_creates_once, CombinedCreate}; +pub use reconstruct::StampedTxstatBlock; pub use resolved_wire::{BlockQueueWaveIntake, ResolvedWire}; pub use soft_densify::{ bq_assign_stop_bytes, soft_assign_restricted, soft_confirm_window_covered, @@ -1019,6 +1020,11 @@ impl Query { &self.store } + /// Stamped confirm-time econ for one create, or `None` if the row is unstamped. + pub fn txstat_row(&self, fk: Fk) -> Result, QueryError> { + self.store.txstat_row(fk) + } + /// Sample-and-reset archived wire-block reconstructs (Esplora `/raw` vs summary). pub fn sample_reset_reconstruct_archived(&self) -> u64 { self.reconstruct_archived.swap(0, AtomicOrdering::Relaxed) @@ -1672,6 +1678,10 @@ impl Query { self.store.get_tx(fk) } + pub fn get_txstat(&self, fk: Fk) -> Result, QueryError> { + self.store.get_txstat(fk) + } + pub fn get_tx_by_txid(&self, txid: &[u8; 32]) -> Result, QueryError> { if let Some(fk) = self.lookup_tx_fk(txid)? { return Ok(Some((fk, self.get_tx(fk)?))); diff --git a/crates/rbitcoin-query/src/query_tests.rs b/crates/rbitcoin-query/src/query_tests.rs index e331b5967..2284e82af 100644 --- a/crates/rbitcoin-query/src/query_tests.rs +++ b/crates/rbitcoin-query/src/query_tests.rs @@ -1,6 +1,6 @@ use super::*; use crate::testutil::FixtureChain; -use rbitcoin_store::{InputRecord, OutputRecord}; +use rbitcoin_store::{InputRecord, OutputRecord, TxRecord, TxStatRow}; #[test] fn query_open_clears_strong_above_tip() { @@ -2630,6 +2630,119 @@ fn reconstruct_pruned_returns_pruned_not_corrupt() { let _ = std::fs::remove_dir_all(&dir); } +fn unstamp_txstat(q: &Query, fk: Fk) { + q.store() + .write_txstat_row( + fk, + &TxStatRow { + n_in: 0, + fee_sat: 0, + base: 0, + wit_extra: 0, + }, + ) + .unwrap(); +} + +#[test] +fn stamp_txstat_from_block_coinbase_and_spend() { + use bitcoin::hashes::Hash; + + let (dir, q) = temp_query("stamp-txstat-from-block"); + let (h0, t0) = coinbase_block(0, Fk::NULL, None); + let hash0 = h0.hash; + let hfk0 = q + .connect_block(Height(0), &h0, std::slice::from_ref(&t0)) + .unwrap(); + let fk0 = q.block_tx_fks(Height(0)).unwrap()[0]; + unstamp_txstat(&q, fk0); + assert!(q.txstat_row(fk0).unwrap().is_none()); + + let empty = bitcoin::Block { + header: bitcoin::block::Header { + version: bitcoin::block::Version::ONE, + prev_blockhash: bitcoin::BlockHash::from_byte_array([0; 32]), + merkle_root: bitcoin::TxMerkleNode::from_byte_array([0; 32]), + time: 1, + bits: bitcoin::CompactTarget::from_consensus(0x207f_ffff), + nonce: 0, + }, + txdata: vec![], + }; + let err = q.stamp_txstat_from_block(Height(99), &empty).unwrap_err(); + assert!( + matches!(err, StoreError::Corrupt(s) if s.contains("stamp txstat missing header")), + "{err:?}" + ); + let err = q.stamp_txstat_from_block(Height(0), &empty).unwrap_err(); + assert!( + matches!(err, StoreError::Corrupt(s) if s.contains("stamp txstat fk count")), + "{err:?}" + ); + + let b0 = q.reconstruct_archived_block(&hash0).unwrap().unwrap(); + q.stamp_txstat_from_block(Height(0), &b0).unwrap(); + let row0 = q.txstat_row(fk0).unwrap().expect("stamped coinbase"); + assert_eq!(row0.n_in, 1); + assert_eq!(row0.fee_sat, 0); + assert_eq!(row0.size() as usize, b0.txdata[0].total_size()); + + let (h1, cb1) = coinbase_block(1, hfk0, Some(hash0)); + let foreign = TxApply { + tx: TxRecord { + txid: [0x11; 32], + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: t0.tx.txid, + create_fk: fk0, + prev_index: 0, + sequence: u32::MAX, + script_sig: vec![], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(49_0000_0000, vec![0x51])], + }; + q.connect_block(Height(1), &h1, &[cb1, foreign]).unwrap(); + let fks1 = q.block_tx_fks(Height(1)).unwrap(); + for &fk in &fks1 { + unstamp_txstat(&q, fk); + } + let b1 = q.reconstruct_archived_block(&h1.hash).unwrap().unwrap(); + q.stamp_txstat_from_block(Height(1), &b1).unwrap(); + let foreign_row = q.txstat_row(fks1[1]).unwrap().expect("stamped spend"); + assert_eq!(foreign_row.n_in, 1); + assert_eq!(foreign_row.fee_sat, 1_0000_0000); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn txstat_row_merges_overflow_via_header_blob() { + let (dir, q) = temp_query("txstat-row-ovf"); + let (h0, t0) = coinbase_block(0, Fk::NULL, None); + let hfk = q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let fk = q.block_tx_fks(Height(0)).unwrap()[0]; + assert!(q.store().txstat_row(fk).unwrap().is_some()); + let fat = TxStatRow { + n_in: 2, + fee_sat: 2_000_000, + base: 400, + wit_extra: 200_000, + }; + q.store() + .write_txstat_block(hfk, fk.get().unwrap(), &[fat]) + .unwrap(); + assert_eq!(q.store().txstat_row(fk).unwrap(), Some(fat)); + unstamp_txstat(&q, fk); + assert_eq!(q.store().txstat_row(fk).unwrap(), None); + let _ = std::fs::remove_dir_all(&dir); +} + #[test] fn reorg_through_pruneheight_refuses() { let (dir, q) = temp_query("reorg-pruneheight"); diff --git a/crates/rbitcoin-query/src/reconstruct.rs b/crates/rbitcoin-query/src/reconstruct.rs index 51fc28629..fa1882a4a 100644 --- a/crates/rbitcoin-query/src/reconstruct.rs +++ b/crates/rbitcoin-query/src/reconstruct.rs @@ -4,6 +4,15 @@ use super::*; use crate::U64Map; use std::time::Instant; +/// Stamped `txstat.body` rows for one confirmed block (every row non-zero). +#[derive(Debug, Clone)] +pub struct StampedTxstatBlock { + pub rec: HeaderRecord, + pub fks: Vec, + pub rows: Vec, + pub n_outs: Vec, +} + impl Query { fn load_body_from_store( &self, @@ -411,6 +420,123 @@ impl Query { Some(h) => Ok(Some(self.reconstruct_block_at_height(h)?)), } } + + /// Dense confirm-time econ for a confirmed height, or `None` if any row is unstamped. + /// + /// Does not read inwit. Missing `header_txs` is `None` (header-only). + pub fn stamped_txstat_block( + &self, + height: Height, + ) -> Result, QueryError> { + let Some((header_fk, rec)) = self.header_at_height(height)? else { + return Ok(None); + }; + let Some((first, n)) = self.store.header_txs.get_range(header_fk)? else { + return Ok(None); + }; + if n == 0 { + return Ok(None); + } + let last = first + .0 + .checked_add(u64::from(n.saturating_sub(1))) + .ok_or(StoreError::Corrupt("invariant: header_txs last fk"))?; + let packed = self.store.txstat_range(header_fk, first.0, last)?; + if packed.len() != n as usize || packed.iter().any(Option::is_none) { + return Ok(None); + } + let fks: Vec = (first.0..=last).map(Fk).collect(); + let loc = self.store.tx_create_loc_range_batch(&fks)?; + if loc.len() != fks.len() { + return Err(StoreError::Corrupt("invariant: loc batch length")); + } + let mut n_outs = Vec::with_capacity(fks.len()); + for p in loc { + let p = p.ok_or(StoreError::NotFound)?; + n_outs.push(p.n_out); + } + Ok(Some(StampedTxstatBlock { + rec, + fks, + rows: packed.into_iter().map(|r| r.unwrap()).collect(), + n_outs, + })) + } + + /// Sum of non-coinbase output values (first fk is coinbase). Reads `txout` only. + pub fn non_coinbase_total_out(&self, fks: &[Fk]) -> Result { + let mut sum = 0i64; + for &fk in fks.iter().skip(1) { + let (_, outs) = self.store.get_tx_meta_and_outputs(fk)?; + for o in outs { + sum = sum.saturating_add(o.value); + } + } + Ok(sum) + } + + /// Fill unstamped `txstat` rows from a reconstructed wire block. + pub fn stamp_txstat_from_block(&self, height: Height, block: &Block) -> Result<(), QueryError> { + let Some((header_fk, _)) = self.header_at_height(height)? else { + return Err(StoreError::Corrupt( + "invariant: stamp txstat missing header", + )); + }; + let fks = self.block_tx_fks(height)?; + if fks.len() != block.txdata.len() { + return Err(StoreError::Corrupt("invariant: stamp txstat fk count")); + } + let mut same_block = std::collections::HashMap::new(); + for tx in &block.txdata { + same_block.insert(tx.compute_txid().to_byte_array(), tx); + } + let mut rows = Vec::with_capacity(fks.len()); + for tx in &block.txdata { + let in_sum = txstat_in_sum_from_block(self, &same_block, tx)?; + rows.push(crate::archive::txstat_row_from_tx(tx, in_sum)?); + } + let first = fks + .first() + .and_then(|fk| fk.get()) + .ok_or(StoreError::Corrupt("invariant: stamp txstat first fk"))?; + self.store.write_txstat_block(header_fk, first, &rows)?; + Ok(()) + } +} + +fn txstat_in_sum_from_block( + query: &Query, + same_block: &std::collections::HashMap<[u8; 32], &bitcoin::Transaction>, + tx: &bitcoin::Transaction, +) -> Result, QueryError> { + if tx.is_coinbase() { + return Ok(None); + } + let mut sum = 0u64; + for inp in &tx.input { + let tid = inp.previous_output.txid.to_byte_array(); + let vout = inp.previous_output.vout; + let val = if let Some(parent) = same_block.get(&tid) { + let o = parent + .output + .get(vout as usize) + .ok_or(StoreError::Corrupt("invariant: stamp same-block vout"))?; + o.value.to_sat() + } else { + let pfk = query + .tx_fk_by_txid(&tid)? + .ok_or(StoreError::Corrupt("invariant: stamp missing prev tx"))?; + let o = query.tx_output_at_fk(pfk, vout)?; + if o.value < 0 { + return Err(StoreError::Corrupt("txstat prevout negative")); + } + o.value as u64 + }; + sum = sum + .checked_add(val) + .ok_or(StoreError::Corrupt("txstat in_sum overflow"))?; + } + Ok(Some(sum)) } fn encode_witness_block( @@ -495,4 +621,63 @@ mod encode_witness_tests { encode_class_a_tx(&mut plain, &rec, &no_wit, &outs); assert_ne!(&plain[4..6], &[0, 1]); } + + #[test] + fn txstat_in_sum_reads_same_block_parent() { + use bitcoin::hashes::Hash; + let (_dir, q) = crate::testutil::tiny_query_labeled("txstat-in-sum-same"); + let parent = bitcoin::Transaction { + version: bitcoin::transaction::Version::ONE, + lock_time: bitcoin::absolute::LockTime::ZERO, + input: vec![bitcoin::TxIn { + previous_output: bitcoin::OutPoint::null(), + script_sig: bitcoin::script::ScriptBuf::from_bytes(vec![0x01]), + sequence: bitcoin::Sequence::MAX, + witness: bitcoin::Witness::new(), + }], + output: vec![bitcoin::TxOut { + value: bitcoin::Amount::from_sat(50_0000_0000), + script_pubkey: bitcoin::script::ScriptBuf::from_bytes(vec![0x51]), + }], + }; + let child = bitcoin::Transaction { + version: bitcoin::transaction::Version::ONE, + lock_time: bitcoin::absolute::LockTime::ZERO, + input: vec![bitcoin::TxIn { + previous_output: bitcoin::OutPoint { + txid: parent.compute_txid(), + vout: 0, + }, + script_sig: bitcoin::script::ScriptBuf::new(), + sequence: bitcoin::Sequence::MAX, + witness: bitcoin::Witness::new(), + }], + output: vec![bitcoin::TxOut { + value: bitcoin::Amount::from_sat(49_0000_0000), + script_pubkey: bitcoin::script::ScriptBuf::from_bytes(vec![0x51]), + }], + }; + let mut same = std::collections::HashMap::new(); + same.insert(parent.compute_txid().to_byte_array(), &parent); + assert_eq!( + txstat_in_sum_from_block(&q, &same, &child).unwrap(), + Some(50_0000_0000) + ); + let bad_vout = bitcoin::Transaction { + input: vec![bitcoin::TxIn { + previous_output: bitcoin::OutPoint { + txid: parent.compute_txid(), + vout: 9, + }, + ..child.input[0].clone() + }], + ..child.clone() + }; + let err = txstat_in_sum_from_block(&q, &same, &bad_vout).unwrap_err(); + assert!( + matches!(err, StoreError::Corrupt(s) if s.contains("stamp same-block vout")), + "{err:?}" + ); + let _ = std::fs::remove_dir_all(_dir.path()); + } } diff --git a/crates/rbitcoin-rpc/src/blockstats.rs b/crates/rbitcoin-rpc/src/blockstats.rs index dd327840d..381b34d58 100644 --- a/crates/rbitcoin-rpc/src/blockstats.rs +++ b/crates/rbitcoin-rpc/src/blockstats.rs @@ -1,18 +1,16 @@ -//! Core `getblockstats` — reconstruct a block and sum fees / UTXO / weight. +//! Core `getblockstats` — stamped `txstat` rows, reconstruct fallback. use crate::methods::{ map_query, parse_hash32_display, rpc_error, RpcContext, RpcParams, ERR_INVALID_ADDRESS_OR_KEY, ERR_INVALID_PARAMETER, ERR_MISC, }; -use bitcoin::consensus::Encodable; use bitcoin::hashes::Hash; -use bitcoin::{Amount, Block, OutPoint, ScriptBuf, TxOut}; +use bitcoin::{Amount, Block, BlockHash, OutPoint, ScriptBuf, TxOut}; use rbitcoin_primitives::Height; +use rbitcoin_query::StampedTxstatBlock; +use rbitcoin_store::{HeaderRecord, TxStatRow}; use serde_json::{json, Map, Value}; -/// Core `PER_UTXO_OVERHEAD` = `sizeof(COutPoint)+sizeof(uint32_t)+sizeof(bool)`. -pub const PER_UTXO_OVERHEAD: i64 = 41; - /// Weight percentiles reported by `getblockstats` (10 / 25 / 50 / 75 / 90). const PERCENTILES: [i64; 5] = [10, 25, 50, 75, 90]; @@ -23,14 +21,6 @@ pub fn is_unspendable(script: &[u8]) -> bool { rbitcoin_consensus::policy::is_unspendable(script) } -/// Consensus-serialized `CTxOut` size (value + compact script). -pub fn txout_serialized_size(out: &TxOut) -> i64 { - let mut buf = Vec::new(); - out.consensus_encode(&mut buf) - .expect("TxOut consensus encode is infallible"); - buf.len() as i64 -} - /// Core `CalculateTruncatedMedian`: even length is the integer mean of the two middles. pub fn truncated_median(mut scores: Vec) -> i64 { let n = scores.len(); @@ -95,9 +85,6 @@ pub struct BlockStats { pub totalfee: i64, pub txs: i64, pub utxo_increase: i64, - pub utxo_size_inc: i64, - pub utxo_increase_actual: i64, - pub utxo_size_inc_actual: i64, } impl BlockStats { @@ -138,15 +125,6 @@ impl BlockStats { m.insert("totalfee".into(), json!(self.totalfee)); m.insert("txs".into(), json!(self.txs)); m.insert("utxo_increase".into(), json!(self.utxo_increase)); - m.insert("utxo_size_inc".into(), json!(self.utxo_size_inc)); - m.insert( - "utxo_increase_actual".into(), - json!(self.utxo_increase_actual), - ); - m.insert( - "utxo_size_inc_actual".into(), - json!(self.utxo_size_inc_actual), - ); m } @@ -183,9 +161,6 @@ pub fn compute_block_stats( ) -> Result { let mut ins = 0i64; let mut outs = 0i64; - let mut utxo_size_inc = 0i64; - let mut utxo_size_inc_actual = 0i64; - let mut utxo_increase_actual = 0i64; let mut total_out = 0i64; let mut total_size = 0i64; let mut total_weight = 0i64; @@ -202,7 +177,6 @@ pub fn compute_block_stats( let mut maxfeerate = 0i64; let mut mintxsize = i64::MAX; let mut maxtxsize = 0i64; - let skip_actual = height == 0; for tx in &block.txdata { let is_cb = tx.is_coinbase(); @@ -224,12 +198,6 @@ pub fn compute_block_stats( let po = prevout(&inp.previous_output) .ok_or_else(|| format!("missing prevout {}", inp.previous_output))?; input_value += po.value.to_sat() as i64; - let ser = txout_serialized_size(&po) + PER_UTXO_OVERHEAD; - utxo_size_inc -= ser; - if !is_unspendable(po.script_pubkey.as_bytes()) { - utxo_size_inc_actual -= ser; - utxo_increase_actual -= 1; - } } let mut output_value = 0i64; for o in &tx.output { @@ -254,15 +222,7 @@ pub fn compute_block_stats( maxtxsize = maxtxsize.max(tx_size); } - for o in &tx.output { - outs += 1; - let ser = txout_serialized_size(o) + PER_UTXO_OVERHEAD; - utxo_size_inc += ser; - if !skip_actual && !is_unspendable(o.script_pubkey.as_bytes()) { - utxo_size_inc_actual += ser; - utxo_increase_actual += 1; - } - } + outs += tx.output.len() as i64; } let n_non_cb = fees.len() as i64; @@ -312,12 +272,121 @@ pub fn compute_block_stats( totalfee, txs: block.txdata.len() as i64, utxo_increase: outs - ins, - utxo_size_inc, - utxo_increase_actual, - utxo_size_inc_actual, }) } +fn stats_from_txstat( + height: u32, + rec: &HeaderRecord, + mediantime: u32, + subsidy: i64, + rows: &[TxStatRow], + n_outs: &[u32], + total_out: i64, +) -> BlockStats { + let mut ins = 0i64; + let mut outs = 0i64; + let mut total_size = 0i64; + let mut total_weight = 0i64; + let mut totalfee = 0i64; + let mut swtotal_size = 0i64; + let mut swtotal_weight = 0i64; + let mut swtxs = 0i64; + let mut fees = Vec::new(); + let mut sizes = Vec::new(); + let mut feerate_weights: Vec<(i64, i64)> = Vec::new(); + let mut minfee = i64::MAX; + let mut maxfee = 0i64; + let mut minfeerate = i64::MAX; + let mut maxfeerate = 0i64; + let mut mintxsize = i64::MAX; + let mut maxtxsize = 0i64; + + for (i, (row, n_out)) in rows.iter().zip(n_outs.iter()).enumerate() { + outs += i64::from(*n_out); + let is_cb = i == 0; + if is_cb { + continue; + } + ins += i64::from(row.n_in); + let tx_size = i64::try_from(row.size()).unwrap_or(i64::MAX); + let tx_weight = i64::try_from(row.weight()).unwrap_or(i64::MAX); + let fee = i64::try_from(row.fee_sat).unwrap_or(i64::MAX); + total_size += tx_size; + total_weight += tx_weight; + totalfee += fee; + let has_wit = row.has_witness(); + if has_wit { + swtxs += 1; + swtotal_size += tx_size; + swtotal_weight += tx_weight; + } + let feerate = if tx_weight > 0 { + fee.saturating_mul(4) / tx_weight + } else { + 0 + }; + fees.push(fee); + sizes.push(tx_size); + feerate_weights.push((feerate, tx_weight)); + minfee = minfee.min(fee); + maxfee = maxfee.max(fee); + minfeerate = minfeerate.min(feerate); + maxfeerate = maxfeerate.max(feerate); + mintxsize = mintxsize.min(tx_size); + maxtxsize = maxtxsize.max(tx_size); + } + + let n_non_cb = fees.len() as i64; + if n_non_cb == 0 { + minfee = 0; + minfeerate = 0; + mintxsize = 0; + } + let avgfee = if n_non_cb > 0 { totalfee / n_non_cb } else { 0 }; + let avgfeerate = if total_weight > 0 { + totalfee.saturating_mul(4) / total_weight + } else { + 0 + }; + let avgtxsize = if n_non_cb > 0 { + total_size / n_non_cb + } else { + 0 + }; + + BlockStats { + avgfee, + avgfeerate, + avgtxsize, + blockhash: BlockHash::from_byte_array(rec.hash).to_string(), + feerate_percentiles: percentiles_by_weight(feerate_weights, total_weight), + height, + ins, + maxfee, + maxfeerate, + maxtxsize, + medianfee: truncated_median(fees), + mediantime, + mediantxsize: truncated_median(sizes), + minfee, + minfeerate, + mintxsize, + outs, + subsidy, + swtotal_size, + swtotal_weight, + swtxs, + time: rec.timestamp, + total_out, + total_size, + total_weight, + totalfee, + txs: rows.len() as i64, + utxo_increase: outs - ins, + } +} + fn help_err() -> Value { rpc_error(ERR_MISC, HELP) } @@ -433,6 +502,33 @@ fn prevout_map_for_block( map } +fn stats_for_stamped( + ctx: &RpcContext, + height: Height, + stamped: &StampedTxstatBlock, +) -> Result { + let mediantime = rbitcoin_consensus::median_time_past(ctx.query.as_ref(), height) + .unwrap_or(stamped.rec.timestamp); + let params = match ctx.chain.as_ref() { + Some(c) => c.params.clone(), + None => rbitcoin_consensus::ChainParams::for_network(ctx.network), + }; + let subsidy = rbitcoin_consensus::block_subsidy(height.0, ¶ms); + let total_out = ctx + .query + .non_coinbase_total_out(&stamped.fks) + .map_err(|e| rpc_error(ERR_MISC, e.to_string()))?; + Ok(stats_from_txstat( + height.0, + &stamped.rec, + mediantime, + subsidy, + &stamped.rows, + &stamped.n_outs, + total_out, + )) +} + fn stats_for_connected( ctx: &RpcContext, height: Height, @@ -467,7 +563,7 @@ pub fn getblockstats(ctx: &RpcContext, params: &RpcParams) -> Result { if h < 0 { return Err(rpc_error( @@ -481,12 +577,7 @@ pub fn getblockstats(ctx: &RpcContext, params: &RpcParams) -> Result { match ctx @@ -494,13 +585,7 @@ pub fn getblockstats(ctx: &RpcContext, params: &RpcParams) -> Result { - let block = ctx - .query - .reconstruct_block_at_height(height) - .map_err(|e| map_query(e, "Block not available (pruned data)"))?; - (height, block) - } + Some(height) => height, None => { if ctx .query @@ -519,6 +604,20 @@ pub fn getblockstats(ctx: &RpcContext, params: &RpcParams) -> Result Result Result>, StoreError> { + let blob = self.txs.txstat.header_blob(header_fk)?; + self.txs.txstat.get_range(first, last, Some(&blob)) + } + + /// Overwrite one existing `txstat` row that fits in 8 B (tests / placeholders). + pub fn write_txstat_row(&self, fk: Fk, row: &crate::TxStatRow) -> Result<(), StoreError> { + self.txs.txstat.write_row(fk, row) + } + + /// Stamp a confirmed block's `txstat` cells and that header's overflow blob. + pub fn write_txstat_block( + &self, + header_fk: Fk, + first_fk: u64, + rows: &[crate::TxStatRow], + ) -> Result<(), StoreError> { + self.txs.txstat.write_block_rows(header_fk, first_fk, rows) + } + + /// One `txstat.body` row, or `None` if unstamped (all-zero). Loads overflow if needed. + pub fn txstat_row(&self, fk: Fk) -> Result, StoreError> { + let cell = self.txs.txstat.get_cell(fk)?; + match crate::txstat::parse_cell(cell)? { + crate::txstat::CellParse::Unstamped => Ok(None), + crate::txstat::CellParse::Complete(row) => Ok(Some(row)), + crate::txstat::CellParse::NeedTail => { + let h = self + .tx_height_get(fk)? + .ok_or(StoreError::Corrupt("invariant: txstat overflow missing"))?; + let hfk = self + .confirmed + .get(Height(h))? + .ok_or(StoreError::Corrupt("invariant: txstat overflow missing"))?; + let (first, _) = self + .header_txs + .get_range(hfk)? + .ok_or(StoreError::Corrupt("invariant: txstat overflow missing"))?; + let blob = self.txs.txstat.header_blob(hfk)?; + self.txs.txstat.get_row_merged(fk, first.0, &blob) + } + } + } + /// Absolute body `(offset, len)` for `fk` (for cache idx cache). pub fn tx_body_range(&self, fk: Fk) -> Result<(u64, u64), StoreError> { self.txs.body_range(fk) @@ -743,10 +793,31 @@ impl Store { .put_full_batch_from_pins(items, index, spent_overlay) } + pub fn put_tx_full_batch_from_pins_with_txstat( + &self, + items: &[(P, Vec)], + index: bool, + spent_overlay: &[Vec<(u32, Fk, u32)>], + txstat: &[crate::txstat::TxStatRow], + header_ranges: &[(Fk, Fk, u32)], + ) -> Result<(Vec, Vec), StoreError> { + self.txs.put_full_batch_from_pins_with_txstat( + items, + index, + spent_overlay, + txstat, + header_ranges, + ) + } + pub fn get_tx_by_txid(&self, txid: &[u8; 32]) -> Result, StoreError> { self.txs.get_by_txid(txid) } + pub fn get_txstat(&self, fk: Fk) -> Result, StoreError> { + self.txstat_row(fk) + } + /// Annotate create outpoint as spent by `spending_tx_fk` at `spending_vin`. pub fn put_spend_create( &self, @@ -2303,6 +2374,40 @@ mod tests { let _ = std::fs::remove_dir_all(&dir2); } + #[test] + fn open_schema24_occupied_creates_zero_txstat() { + let dir = tmp(); + { + let s = Store::create_tiny(&dir).unwrap(); + let item = coinbase_item([0x25u8; 32], vec![OutputRecord::unspent(50, vec![0x51])]); + s.put_tx_full_batch_indexed(&[item], true).unwrap(); + s.flush().unwrap(); + } + std::fs::remove_file(dir.join("txstat.body")).unwrap(); + write_store_meta_ver(&dir, 24); + let s = Store::open_tiny(&dir).unwrap(); + assert_eq!(s.txs.txstat.count(), 1); + assert_eq!(s.txs.txstat.get_cell(Fk(1)).unwrap(), [0u8; 8]); + drop(s); + assert_eq!(read_store_meta_ver(&dir), SCHEMA_VERSION); + assert!(dir.join("txstat.body").is_file()); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn check_meta_refuses_schema_past_this_binary() { + let dir = tmp(); + std::fs::create_dir_all(&dir).unwrap(); + let mut bytes = STORE_MAGIC.to_vec(); + bytes.extend_from_slice(&(SCHEMA_VERSION + 1).to_le_bytes()); + std::fs::write(dir.join("meta"), bytes).unwrap(); + assert!(matches!( + check_meta(&dir), + Err(StoreError::BadSchema(v)) if v == SCHEMA_VERSION + 1 + )); + let _ = std::fs::remove_dir_all(&dir); + } + fn write_store_meta_ver(dir: &Path, ver: u16) { let mut bytes = STORE_MAGIC.to_vec(); bytes.extend_from_slice(&ver.to_le_bytes()); @@ -2332,7 +2437,7 @@ mod tests { "schema 22 open must keep create.loc" ); assert_eq!(read_store_meta_ver(&dir), SCHEMA_VERSION); - assert_eq!(SCHEMA_VERSION, 24); + assert_eq!(SCHEMA_VERSION, 25); let s = Store::open_tiny(&dir).unwrap(); drop(s); assert_eq!(read_store_meta_ver(&dir), SCHEMA_VERSION); @@ -2477,7 +2582,7 @@ mod tests { let s = Store::open_tiny(&dir).unwrap(); drop(s); assert_eq!(read_store_meta_ver(&dir), SCHEMA_VERSION); - assert_eq!(SCHEMA_VERSION, 24); + assert_eq!(SCHEMA_VERSION, 25); assert!( !dir.join("spent.off").exists(), "empty 21 open must unlink leftover spent.off" diff --git a/crates/rbitcoin-store/src/tx_table/mod.rs b/crates/rbitcoin-store/src/tx_table/mod.rs index aef6c013b..b86c6b5a9 100644 --- a/crates/rbitcoin-store/src/tx_table/mod.rs +++ b/crates/rbitcoin-store/src/tx_table/mod.rs @@ -115,12 +115,15 @@ const BODY_META_V17_VER_1: u8 = 1 << 0; const BODY_META_V17_VER_2: u8 = 1 << 1; const BODY_META_V17_VER_3: u8 = 1 << 2; const BODY_META_V17_LOCKTIME_ZERO: u8 = 1 << 3; -const BODY_META_V17_RESERVED: u8 = 0x70; +/// `n_in` lives on `txstat.body`; no uleb `input_count` follows locktime. +pub(crate) const BODY_META_V17_N_IN_TXSTAT: u8 = 1 << 4; +const BODY_META_V17_RESERVED: u8 = 0x60; const BODY_META_V17_VER_MASK: u8 = BODY_META_V17_VER_1 | BODY_META_V17_VER_2 | BODY_META_V17_VER_3; -/// Encode schema-17 thin meta. Production still writes [`TxRecord::encode_body_meta_into`]. +/// Encode schema-17 thin meta. New writes set [`BODY_META_V17_N_IN_TXSTAT`] +/// and omit the `input_count` uleb (`n_in` is on `txstat.body`). pub(crate) fn encode_body_meta_v17(rec: &TxRecord, out: &mut Vec) { - let mut flags = BODY_META_V17_LAYOUT17; + let mut flags = BODY_META_V17_LAYOUT17 | BODY_META_V17_N_IN_TXSTAT; match rec.version { 1 => flags |= BODY_META_V17_VER_1, 2 => flags |= BODY_META_V17_VER_2, @@ -137,7 +140,6 @@ pub(crate) fn encode_body_meta_v17(rec: &TxRecord, out: &mut Vec) { if rec.locktime != 0 { write_uleb128(out, u64::from(rec.locktime)); } - write_uleb128(out, u64::from(rec.input_count)); } /// Decode schema-17 thin meta. Rejects schema-15 16-byte prefixes (no LAYOUT17 bit). @@ -183,18 +185,23 @@ pub(crate) fn decode_body_meta_v17(buf: &[u8]) -> Result<(TxRecord, usize), Stor off += n; v as u32 }; - let (nin, n1) = read_uleb128(&buf[off..])?; - if nin > u64::from(u32::MAX) { - return Err(StoreError::Corrupt("v17 input_count overflow")); - } - off += n1; + let input_count = if flags & BODY_META_V17_N_IN_TXSTAT != 0 { + 0 + } else { + let (nin, n1) = read_uleb128(&buf[off..])?; + if nin > u64::from(u32::MAX) { + return Err(StoreError::Corrupt("v17 input_count overflow")); + } + off += n1; + nin as u32 + }; Ok(( TxRecord { txid: [0u8; 32], version, locktime, input_start_fk: Fk::NULL, - input_count: nin as u32, + input_count, output_start_fk: Fk::NULL, output_count: 0, }, @@ -202,6 +209,24 @@ pub(crate) fn decode_body_meta_v17(buf: &[u8]) -> Result<(TxRecord, usize), Stor )) } +fn txstat_placeholder(n_in: u32) -> crate::txstat::TxStatRow { + crate::txstat::TxStatRow { + n_in, + fee_sat: 0, + base: 0, + wit_extra: 0, + } +} + +fn txstat_placeholders( + items: &[(TxRecord, Vec, Vec)], +) -> Vec { + items + .iter() + .map(|(tx, _, _)| txstat_placeholder(tx.input_count)) + .collect() +} + /// Class A output (addressed via `tx.output_start_fk` run + local vout). #[derive(Clone, Debug, PartialEq, Eq)] pub struct OutputRecord { @@ -407,6 +432,8 @@ pub struct TxTable { pub(crate) head: SegmentedTxHead, /// Dense create_fk-ordered txids (schema 13+). pub(crate) txids: crate::txid_body::TxidBody, + /// Dense create_fk-ordered confirm-time econ (schema 25). + pub(crate) txstat: crate::txstat::TxStat, /// Datadir secret: keyed head probes + script XOR (schema 12+). pub(crate) secret: crate::store_secret::StoreSecret, /// Unflushed head inserts (write-behind). Readers see published snapshot. @@ -478,77 +505,107 @@ fn unlink_leftover_class_a_idx(dir: &Path) -> Result<(), StoreError> { Ok(()) } -fn repair_class_a_count_skew( - create_loc: &crate::create_loc::CreateLoc, - inwit_loc: &crate::delta_loc::DeltaLoc, - body: &VarTable, - spent: &VarTable, - inwit: &VarTable, - txids: &crate::txid_body::TxidBody, +struct ClassASkewStems<'a> { + create_loc: &'a crate::create_loc::CreateLoc, + inwit_loc: &'a crate::delta_loc::DeltaLoc, + body: &'a VarTable, + spent: &'a VarTable, + inwit: &'a VarTable, + txids: &'a crate::txid_body::TxidBody, + txstat: &'a crate::txstat::TxStat, +} + +fn class_a_skew_target_count( + n_loc: u64, + n_txids: u64, + n_inwit_loc: u64, prune_inwit_mode: bool, -) -> Result<(), StoreError> { - let n_loc = create_loc.count(); - let n_txids = txids.count(); - let n_inwit_loc = inwit_loc.count(); +) -> Option { if n_txids == n_loc && (prune_inwit_mode || n_inwit_loc == n_loc) { - return Ok(()); + return None; } - let n = if prune_inwit_mode { + Some(if prune_inwit_mode { n_loc.min(n_txids) } else { n_loc.min(n_txids).min(n_inwit_loc) - }; - rbitcoin_log::warn!( - "store: Class A count skew loc={n_loc} inwit.loc={n_inwit_loc} \ - txid.body={n_txids} — truncating to {n}" - ); - let (tx_end, sp_end, in_end) = if n == 0 { + }) +} + +fn class_a_skew_stem_ends( + stems: &ClassASkewStems<'_>, + n: u64, + prune_inwit_mode: bool, +) -> Result<(u64, u64, u64), StoreError> { + if n == 0 { let h = crate::file::FILE_HEADER_LEN as u64; - (h, h, h) + return Ok((h, h, h)); + } + let p = stems + .create_loc + .range_batch(&[Fk(n)])? + .into_iter() + .next() + .flatten() + .ok_or(StoreError::Corrupt("invariant: loc range for truncate"))?; + let in_end = if prune_inwit_mode { + crate::file::FILE_HEADER_LEN as u64 } else { - let p = create_loc + let ir = stems + .inwit_loc .range_batch(&[Fk(n)])? .into_iter() .next() .flatten() - .ok_or(StoreError::Corrupt("invariant: loc range for truncate"))?; - let in_end = if prune_inwit_mode { - crate::file::FILE_HEADER_LEN as u64 - } else { - let ir = inwit_loc - .range_batch(&[Fk(n)])? - .into_iter() - .next() - .flatten() - .ok_or(StoreError::Corrupt( - "invariant: inwit.loc range for truncate", - ))?; - ir.0.saturating_add(ir.1) - }; - ( - p.txout.0.saturating_add(p.txout.1), - p.spent.0.saturating_add(p.spent.1), - in_end, - ) + .ok_or(StoreError::Corrupt( + "invariant: inwit.loc range for truncate", + ))?; + ir.0.saturating_add(ir.1) }; - create_loc.truncate_to_count(n)?; + Ok(( + p.txout.0.saturating_add(p.txout.1), + p.spent.0.saturating_add(p.spent.1), + in_end, + )) +} + +fn class_a_skew_apply_truncate( + stems: &ClassASkewStems<'_>, + n: u64, + n_txids: u64, + tx_end: u64, + sp_end: u64, + in_end: u64, + prune_inwit_mode: bool, +) -> Result<(), StoreError> { + stems.create_loc.truncate_to_count(n)?; if !prune_inwit_mode { - inwit_loc.truncate_to_count(n)?; + stems.inwit_loc.truncate_to_count(n)?; } - body.truncate_body_to(n, tx_end)?; - spent.truncate_body_to(n, sp_end)?; + stems.body.truncate_body_to(n, tx_end)?; + stems.spent.truncate_body_to(n, sp_end)?; if !prune_inwit_mode { - inwit.truncate_body_to(n, in_end)?; + stems.inwit.truncate_body_to(n, in_end)?; } if n_txids > n { - txids.truncate_to_count(n)?; + stems.txids.truncate_to_count(n)?; + } + if stems.txstat.count() > n { + stems.txstat.truncate_to_count(n)?; } - if body.count() != txids.count() || create_loc.count() != txids.count() { + Ok(()) +} + +fn class_a_skew_assert_aligned( + stems: &ClassASkewStems<'_>, + prune_inwit_mode: bool, +) -> Result<(), StoreError> { + if stems.body.count() != stems.txids.count() || stems.create_loc.count() != stems.txids.count() + { return Err(StoreError::Corrupt( "Class A stem counts still mismatch after repair (reindex required)", )); } - if !prune_inwit_mode && inwit_loc.count() != txids.count() { + if !prune_inwit_mode && stems.inwit_loc.count() != stems.txids.count() { return Err(StoreError::Corrupt( "Class A stem counts still mismatch after repair (reindex required)", )); @@ -556,6 +613,25 @@ fn repair_class_a_count_skew( Ok(()) } +fn repair_class_a_count_skew( + stems: ClassASkewStems<'_>, + prune_inwit_mode: bool, +) -> Result<(), StoreError> { + let n_loc = stems.create_loc.count(); + let n_txids = stems.txids.count(); + let n_inwit_loc = stems.inwit_loc.count(); + let Some(n) = class_a_skew_target_count(n_loc, n_txids, n_inwit_loc, prune_inwit_mode) else { + return Ok(()); + }; + rbitcoin_log::warn!( + "store: Class A count skew loc={n_loc} inwit.loc={n_inwit_loc} \ + txid.body={n_txids} — truncating to {n}" + ); + let (tx_end, sp_end, in_end) = class_a_skew_stem_ends(&stems, n, prune_inwit_mode)?; + class_a_skew_apply_truncate(&stems, n, n_txids, tx_end, sp_end, in_end, prune_inwit_mode)?; + class_a_skew_assert_aligned(&stems, prune_inwit_mode) +} + impl TxTable { pub fn create(dir: &Path) -> Result { Self::create_with_opts(dir, HeadOpenOpts::MAINNET) @@ -607,6 +683,7 @@ impl TxTable { inwit_loc: crate::delta_loc::DeltaLoc::create(inwit_dir, "inwit")?, head: SegmentedTxHead::create(dir, layout)?, txids: crate::txid_body::TxidBody::create(dir)?, + txstat: crate::txstat::TxStat::create(dir)?, secret, pending_head: pending_head::PendingHeadInserts::new(), rebuild_seal_bits: seal_bits, @@ -712,16 +789,28 @@ impl TxTable { } else { crate::txid_body::TxidBody::create(dir)? }; + let txstat = crate::txstat::TxStat::open(dir)?; repair_class_a_count_skew( - &create_loc, - &inwit_loc, - &body, - &spent, - &inwit, - &txids, + ClassASkewStems { + create_loc: &create_loc, + inwit_loc: &inwit_loc, + body: &body, + spent: &spent, + inwit: &inwit, + txids: &txids, + txstat: &txstat, + }, prune_inwit_mode, )?; let n_bodies = create_loc.count(); + if txstat.count() != n_bodies { + rbitcoin_log::warn!( + "store: txstat.body count={} loc={} — aligning to loc", + txstat.count(), + n_bodies + ); + txstat.extend_or_truncate_to(n_bodies)?; + } let mut need_rebuild = false; let head = if !crate::segmented_head::head_meta_exists(dir) { need_rebuild = n_bodies > 0; @@ -791,12 +880,16 @@ impl TxTable { inwit_loc, head, txids, + txstat, secret, pending_head: pending_head::PendingHeadInserts::new(), rebuild_seal_bits: seal_bits, rebuild_workers: workers, prune_inwit_mode: std::sync::atomic::AtomicBool::new(prune_inwit_mode), }; + if n_bodies > 0 { + let _ = t.txstat.n_in_at(Fk(1))?; + } if need_rebuild { let bits = t.head_bits(); let slots = t.head_slots(); @@ -1101,27 +1194,49 @@ impl TxTable { /// Meta + input prevouts only (no script/witness allocation, no outputs). /// - /// Used by load: discover parents without full parse into RAM. + /// Stamped `txstat` supplies `n_in` so this path is 8 B + inwit (no txout). + /// Leftover unstamped rows still decode LAYOUT17 uleb from `txout`. pub fn get_meta_and_prevouts(&self, fk: Fk) -> Result<(TxRecord, Vec<(Fk, u32)>), StoreError> { if self.prune_inwit_mode() { return Err(StoreError::NotFound); } - let mut tx = self.get(fk)?; - let inwit = { - let ir = self - .inwit_loc - .range_batch(&[fk])? - .into_iter() - .next() - .flatten() - .ok_or(StoreError::NotFound)?; - self.inwit.with_bytes_at(ir.0, ir.1, |b| Ok(b.to_vec()))? + let ir = self + .inwit_loc + .range_batch(&[fk])? + .into_iter() + .next() + .flatten() + .ok_or(StoreError::NotFound)?; + let inwit = self.inwit.with_bytes_at(ir.0, ir.1, |b| Ok(b.to_vec()))?; + let mut tx = if let Some(n_in) = self.txstat.n_in_at(fk)? { + TxRecord { + txid: [0u8; 32], + version: 0, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: n_in, + output_start_fk: Fk::NULL, + output_count: 0, + } + } else { + self.get(fk)? }; let prevs = scan_inwit_prevouts(&inwit, tx.input_count)?; tx.txid = self.txids.get(fk)?; Ok((tx, prevs)) } + pub(crate) fn overlay_stamped_n_in(&self, fk: Fk, tx: &mut TxRecord) -> Result<(), StoreError> { + let Some(n_in) = self.txstat.n_in_at(fk)? else { + return Ok(()); + }; + if tx.input_count != 0 && tx.input_count != n_in { + return Err(StoreError::Corrupt("txstat n_in mismatch txout")); + } + tx.input_count = n_in; + Ok(()) + } + pub fn reserve_append(&self, body_bytes: u64, n_records: u64) -> Result<(), StoreError> { self.body.reserve_append(body_bytes, n_records) } @@ -1139,6 +1254,7 @@ impl TxTable { let (mut tx, _, _, _) = decode_packed_tx_with_spender_rels_secret(&raw, pair.n_out, Some(&self.secret))?; tx.txid = self.txids.get(fk)?; + self.overlay_stamped_n_in(fk, &mut tx)?; Ok(tx) } @@ -1277,6 +1393,7 @@ impl TxTable { ) { Ok((mut tx, live, sparse)) => { tx.txid = *known_txid; + self.overlay_stamped_n_in(*fk, &mut tx)?; out.push(Some((tx, live, sparse))); } Err(StoreError::NotFound) | Err(StoreError::Corrupt(_)) => out.push(None), @@ -1669,7 +1786,12 @@ impl TxTable { .flatten() .ok_or(StoreError::NotFound)?; let inwit = self.inwit.with_bytes_at(ir.0, ir.1, |b| Ok(b.to_vec()))?; - let ins = decode_inwit_secret(&inwit, tx.input_count, Some(&self.secret))?; + let ins = if tx.input_count == 0 { + decode_inwit_secret_to_end(&inwit, Some(&self.secret))? + } else { + decode_inwit_secret(&inwit, tx.input_count, Some(&self.secret))? + }; + tx.input_count = ins.len() as u32; tx.txid = self.txids.get(fk)?; Ok((tx, ins, outs)) } @@ -1730,7 +1852,12 @@ impl TxTable { let iraw = span_rec(&inwit_span, i0, ioff, ilen)?; let (mut tx, _ins, outs, _) = decode_packed_tx_with_spender_rels_secret(traw, n_outs[i], Some(&self.secret))?; - let ins = decode_inwit_secret(iraw, tx.input_count, Some(&self.secret))?; + let ins = if tx.input_count == 0 { + decode_inwit_secret_to_end(iraw, Some(&self.secret))? + } else { + decode_inwit_secret(iraw, tx.input_count, Some(&self.secret))? + }; + tx.input_count = ins.len() as u32; tx.txid = ids[i]; out.push((tx, ins, outs)); } @@ -1754,6 +1881,7 @@ impl TxTable { let (mut tx, outs, _) = decode_packed_tx_outs_with_spender_rels_secret(&raw, pair.n_out, Some(&self.secret))?; tx.txid = self.txids.get(fk)?; + self.overlay_stamped_n_in(fk, &mut tx)?; Ok((tx, outs)) } @@ -1825,15 +1953,28 @@ impl TxTable { Ok(()) } - /// Append Class A rows: `txout` + `inwit` + zero `spent` + `txid.body`. + /// Append Class A rows: `txout` + `inwit` + zero `spent` + `txid.body` + `txstat.body`. pub fn put_full_batch_indexed( &self, items: &[(TxRecord, Vec, Vec)], index: bool, + ) -> Result, StoreError> { + let rows = txstat_placeholders(items); + self.put_full_batch_indexed_with_txstat(items, index, &rows) + } + + pub fn put_full_batch_indexed_with_txstat( + &self, + items: &[(TxRecord, Vec, Vec)], + index: bool, + txstat: &[crate::txstat::TxStatRow], ) -> Result, StoreError> { if items.is_empty() { return Ok(Vec::new()); } + if txstat.len() != items.len() { + return Err(StoreError::Corrupt("txstat batch length")); + } let est_out: usize = items .iter() .map(|(_tx, _ins, outs)| { @@ -1849,7 +1990,10 @@ impl TxTable { .map(|(_tx, _ins, outs)| 16 + outs.len() * OutputRecord::SPENT_SLOT_LEN) .sum(); let base = self.body.count(); - if (!self.prune_inwit_mode() && self.inwit.count() != base) || self.spent.count() != base { + if (!self.prune_inwit_mode() && self.inwit.count() != base) + || self.spent.count() != base + || self.txstat.count() != base + { return Err(StoreError::Corrupt("Class A stem count mismatch on append")); } if items.iter().any(|(_, _, outs)| outs.is_empty()) { @@ -1877,6 +2021,10 @@ impl TxTable { )?; let ids: Vec<[u8; 32]> = items.iter().map(|(tx, _, _)| tx.txid).collect(); self.txids.append_batch(base, &ids)?; + let tails = self.txstat.append_batch(base, txstat)?; + if !tails.is_empty() { + return Err(StoreError::Corrupt("txstat overflow needs header blob")); + } if index { let heads: Vec<([u8; 32], Fk)> = items .iter() @@ -1899,10 +2047,28 @@ impl TxTable { items: &[(P, Vec)], index: bool, spent_overlay: &[Vec<(u32, Fk, u32)>], + ) -> Result<(Vec, Vec), StoreError> { + let rows: Vec = items + .iter() + .map(|(pin, _)| txstat_placeholder(pin.packed_tx().input_count)) + .collect(); + self.put_full_batch_from_pins_with_txstat(items, index, spent_overlay, &rows, &[]) + } + + pub fn put_full_batch_from_pins_with_txstat( + &self, + items: &[(P, Vec)], + index: bool, + spent_overlay: &[Vec<(u32, Fk, u32)>], + txstat: &[crate::txstat::TxStatRow], + header_ranges: &[(Fk, Fk, u32)], ) -> Result<(Vec, Vec), StoreError> { if items.is_empty() { return Ok((Vec::new(), Vec::new())); } + if txstat.len() != items.len() { + return Err(StoreError::Corrupt("txstat batch length")); + } if !spent_overlay.is_empty() && spent_overlay.len() != items.len() { return Err(StoreError::Corrupt("spent overlay length")); } @@ -1916,7 +2082,10 @@ impl TxTable { .map(|(pin, _ins)| 16 + spent_record_len(pin.packed_n_out()) as usize) .sum(); let base = self.body.count(); - if (!self.prune_inwit_mode() && self.inwit.count() != base) || self.spent.count() != base { + if (!self.prune_inwit_mode() && self.inwit.count() != base) + || self.spent.count() != base + || self.txstat.count() != base + { return Err(StoreError::Corrupt("Class A stem count mismatch on append")); } for (i, (pin, _)) in items.iter().enumerate() { @@ -1954,6 +2123,9 @@ impl TxTable { )?; let ids: Vec<[u8; 32]> = items.iter().map(|(pin, _)| pin.packed_txid()).collect(); self.txids.append_batch(base, &ids)?; + let tails = self.txstat.append_batch(base, txstat)?; + self.txstat + .put_overflows_for_headers(header_ranges, &tails)?; if index { let heads: Vec<([u8; 32], Fk)> = items .iter() diff --git a/crates/rbitcoin-store/src/tx_table/packed.rs b/crates/rbitcoin-store/src/tx_table/packed.rs index 8a1457dce..b30dd3ec7 100644 --- a/crates/rbitcoin-store/src/tx_table/packed.rs +++ b/crates/rbitcoin-store/src/tx_table/packed.rs @@ -684,7 +684,7 @@ pub fn decode_inwit_secret( secret: Option<&crate::store_secret::StoreSecret>, ) -> Result, StoreError> { if in_count == 0 { - return Ok(Vec::new()); + return decode_inwit_secret_to_end(raw, secret); } let (mut inputs, used) = decode_input_run_prefix(raw, in_count)?; check_trailing_zero_pad(raw, used)?; @@ -701,6 +701,34 @@ pub fn decode_inwit_secret( Ok(inputs) } +pub(crate) fn decode_inwit_secret_to_end( + raw: &[u8], + secret: Option<&crate::store_secret::StoreSecret>, +) -> Result, StoreError> { + let mut inputs = Vec::new(); + let mut off = 0usize; + loop { + if off >= raw.len() || raw[off..].iter().all(|&b| b == 0) { + break; + } + let (rec, used) = InputRecord::decode_at(&raw[off..])?; + off += used; + inputs.push(rec); + } + check_trailing_zero_pad(raw, off)?; + if let Some(sec) = secret { + for inp in &mut inputs { + if !inp.script_sig.is_empty() { + sec.xor_bytes(0, &mut inp.script_sig); + } + for (wi, item) in inp.witness.iter_mut().enumerate() { + sec.xor_bytes(u64::from(wi as u32).saturating_add(1) << 16, item); + } + } + } + Ok(inputs) +} + /// After walking a packed payload to `logical_end`, accept only zero pad to `raw.len()`. #[inline] pub(super) fn check_trailing_zero_pad(raw: &[u8], logical_end: usize) -> Result<(), StoreError> { @@ -760,6 +788,7 @@ pub fn scan_inwit_prevouts(raw: &[u8], in_count: u32) -> Result, off += used; prevouts.push((create_fk, prev_index)); } + check_trailing_zero_pad(raw, off)?; Ok(prevouts) } @@ -1030,6 +1059,17 @@ mod scan_p2tr_tests { ); } + #[test] + fn decode_inwit_secret_zero_count_walks_payload() { + let ins = vec![InputRecord::coinbase(u32::MAX, vec![0x01, 0x02], vec![])]; + let mut raw = Vec::new(); + encode_inwit_with_secret(&ins, &mut raw, None); + let got = decode_inwit_secret(&raw, 0, None).unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0].script_sig, vec![0x01, 0x02]); + assert!(got[0].is_coinbase()); + } + fn three_out_packed() -> (Vec, usize) { let tx = TxRecord { txid: [0x11u8; 32], diff --git a/crates/rbitcoin-store/src/tx_table/tests.rs b/crates/rbitcoin-store/src/tx_table/tests.rs index 1e095a9ff..88418f569 100644 --- a/crates/rbitcoin-store/src/tx_table/tests.rs +++ b/crates/rbitcoin-store/src/tx_table/tests.rs @@ -170,6 +170,164 @@ fn put_full_batch_from_pins_roundtrip() { let _ = std::fs::remove_dir_all(&dir); } +#[test] +fn put_full_batch_writes_txstat_and_reopen() { + let dir = tempfile_dir("txstat-append"); + let t = create_tiny(&dir); + let tx = TxRecord { + txid: [9u8; 32], + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }; + let ins = vec![InputRecord::coinbase(u32::MAX, vec![0x01], vec![])]; + let outs = vec![OutputRecord::unspent(7, vec![0x51])]; + let fks = t.put_full_batch_indexed(&[(tx, ins, outs)], true).unwrap(); + assert_eq!(fks, vec![Fk(1)]); + assert_eq!(t.txstat.count(), 1); + assert_eq!( + crate::txstat::parse_cell(t.txstat.get_cell(Fk(1)).unwrap()).unwrap(), + crate::txstat::CellParse::Complete(crate::txstat::TxStatRow { + n_in: 1, + fee_sat: 0, + base: 0, + wit_extra: 0, + }) + ); + drop(t); + let t2 = TxTable::open_tiny(&dir).unwrap(); + assert_eq!(t2.count(), 1); + assert_eq!(t2.txid_sidefile().count(), 1); + assert_eq!(t2.txstat.count(), 1); + assert_eq!( + crate::txstat::parse_cell(t2.txstat.get_cell(Fk(1)).unwrap()).unwrap(), + crate::txstat::CellParse::Complete(crate::txstat::TxStatRow { + n_in: 1, + fee_sat: 0, + base: 0, + wit_extra: 0, + }) + ); + let _ = std::fs::remove_dir_all(&dir); +} + +fn two_input_item() -> (TxRecord, Vec, Vec) { + let tx = TxRecord { + txid: [0x11u8; 32], + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 2, + output_start_fk: Fk::NULL, + output_count: 1, + }; + let ins = vec![ + InputRecord::coinbase(u32::MAX, vec![0x01], vec![]), + InputRecord { + prev_txid: [2u8; 32], + create_fk: Fk(1), + prev_index: 0, + sequence: u32::MAX, + script_sig: vec![], + witness: vec![], + }, + ]; + let outs = vec![OutputRecord::unspent(7, vec![0x51])]; + (tx, ins, outs) +} + +#[test] +fn prevouts_use_txstat_n_in_without_full_txout() { + let dir = tempfile_dir("prevouts-txstat"); + let t = create_tiny(&dir); + let fk = t.put_full_batch_indexed(&[two_input_item()], true).unwrap()[0]; + let (off, len) = t.body_range(fk).unwrap(); + t.body + .write_body_abs(off, &vec![0u8; len as usize]) + .unwrap(); + assert!(t.get(fk).is_err(), "txout zeros must fail full get"); + let (meta, prevs) = t.get_meta_and_prevouts(fk).unwrap(); + assert_eq!(meta.input_count, 2); + assert_eq!(prevs.len(), 2); + assert_eq!(prevs[0], (Fk::NULL, u32::MAX)); + assert_eq!(prevs[1], (Fk(1), 0)); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn new_create_txout_meta_omits_n_in() { + let dir = tempfile_dir("txout-omit-nin"); + let t = create_tiny(&dir); + let tx = TxRecord { + txid: [4u8; 32], + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }; + let ins = vec![InputRecord::coinbase(u32::MAX, vec![0x01], vec![])]; + let outs = vec![OutputRecord::unspent(7, vec![0x51])]; + let pin = std::sync::Arc::new((tx.clone(), outs.clone())); + let mut pin_bytes = Vec::new(); + pin.encode_txout_body(&mut pin_bytes, None); + assert_eq!( + pin_bytes[0] & BODY_META_V17_N_IN_TXSTAT, + BODY_META_V17_N_IN_TXSTAT, + "pin first-wave txout omits n_in" + ); + let fk = t.put_full_batch_indexed(&[(tx, ins, outs)], true).unwrap()[0]; + let (off, len) = t.body_range(fk).unwrap(); + let raw = t.with_body_span(off, len, |b| Ok(b.to_vec())).unwrap(); + assert_eq!( + raw[0] & BODY_META_V17_N_IN_TXSTAT, + BODY_META_V17_N_IN_TXSTAT + ); + let (meta, n) = decode_body_meta_v17(&raw).unwrap(); + assert_eq!(n, 1, "v1 locktime 0 omit n_in is one flag byte"); + assert_eq!(meta.input_count, 0); + let (got, gouts) = t.get_meta_and_outputs(fk).unwrap(); + assert_eq!(got.input_count, 1, "stamped txstat fills n_in"); + assert_eq!(gouts.len(), 1); + let leftover = [0x89u8, 0x01]; + let (old, on) = decode_body_meta_v17(&leftover).unwrap(); + assert_eq!(on, 2); + assert_eq!(old.input_count, 1); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn txstat_n_in_mismatch_inwit_is_corrupt() { + let dir = tempfile_dir("txstat-nin-mismatch"); + let t = create_tiny(&dir); + let fk = t.put_full_batch_indexed(&[two_input_item()], true).unwrap()[0]; + t.txstat + .write_row( + fk, + &crate::txstat::TxStatRow { + n_in: 1, + fee_sat: 0, + base: 0, + wit_extra: 0, + }, + ) + .unwrap(); + match t.get_meta_and_prevouts(fk) { + Err(StoreError::Corrupt(m)) => { + assert!( + m.contains("trailing") || m.contains("short") || m.contains("mismatch"), + "{m}" + ); + } + other => panic!("expected Corrupt, got {other:?}"), + } + let _ = std::fs::remove_dir_all(&dir); +} + /// Class A append submits txout+inwit+spent bodies as one pwrite wave (not 3 serial). #[test] fn put_full_batch_one_body_write_wave() { @@ -421,6 +579,7 @@ fn open_repairs_body_leading_txid_count() { let t2 = TxTable::open_tiny(&dir).expect("open should repair skew"); assert_eq!(t2.count(), 3); assert_eq!(t2.txid_sidefile().count(), 3); + assert_eq!(t2.txstat.count(), 3); // Kept prefix still readable. let tx = t2.get(Fk(1)).unwrap(); assert_eq!(tx.txid[0], 1); @@ -523,9 +682,10 @@ fn scan_packed_meta_and_prevouts_no_output_alloc() { encode_packed_tx(&tx, &inputs, &outputs, &mut raw); let (meta, _) = TxRecord::decode_body_meta(&raw).unwrap(); assert_eq!(meta.txid, [0u8; 32], "body scan has no leading txid"); + assert_eq!(meta.input_count, 0, "txout meta omits n_in"); let mut inwit = Vec::new(); encode_inwit_with_secret(&inputs, &mut inwit, None); - let prevouts = scan_inwit_prevouts(&inwit, meta.input_count).unwrap(); + let prevouts = scan_inwit_prevouts(&inwit, tx.input_count).unwrap(); assert_eq!(prevouts.len(), 2); assert_eq!(prevouts[0], (Fk::NULL, u32::MAX)); assert_eq!(prevouts[1], (Fk(1), 1)); @@ -1236,6 +1396,7 @@ fn get_outs_by_range_batch_skips_extend_when_need_in_first_page() { assert_eq!(guess_full_n, 0); let (got, live, sparse) = rows[0].as_ref().expect("range denserels"); assert_eq!(got.txid, txid); + assert_eq!(got.input_count, 1); assert_eq!(live.len(), 1); assert_eq!(live[0].0, 0); assert_eq!(sparse.len(), 1); @@ -1785,7 +1946,10 @@ fn tx_fixed_roundtrip() { }; let enc = rec.encode(); assert!(enc.len() > 32, "txid + thin meta"); - assert_eq!(TxRecord::decode(&enc).unwrap(), rec); + let mut got = TxRecord::decode(&enc).unwrap(); + assert_eq!(got.input_count, 0, "encode omits n_in"); + got.input_count = rec.input_count; + assert_eq!(got, rec); } #[test] @@ -1817,12 +1981,12 @@ fn packed_tx_roundtrip() { assert!(enc.len() >= 3, "thin LAYOUT17 meta"); let (dtx, douts, _) = decode_packed_tx_outs_with_spender_rels(&enc, 2).unwrap(); assert_eq!(dtx.txid, [0u8; 32], "body decode leaves txid zero"); - assert_eq!(dtx.input_count, 1); + assert_eq!(dtx.input_count, 0, "txout meta omits n_in"); assert_eq!(dtx.output_count, 2); assert!(dtx.input_start_fk.get().is_none()); let mut inwit = Vec::new(); encode_inwit_with_secret(&inputs, &mut inwit, None); - let dins = decode_inwit_secret(&inwit, dtx.input_count, None).unwrap(); + let dins = decode_inwit_secret(&inwit, tx.input_count, None).unwrap(); assert_eq!(dins, inputs); assert_eq!(douts, outputs); } @@ -1860,11 +2024,12 @@ fn inwit_and_txout_secret_xor_roundtrip() { let (dtx, douts, _) = decode_packed_tx_outs_with_spender_rels(&txout, 1).unwrap(); // Without secret, script stays obfuscated. assert_ne!(douts[0].script, outputs[0].script); + assert_eq!(dtx.input_count, 0, "txout meta omits n_in"); let (dtx2, douts2, _) = decode_packed_tx_outs_with_spender_rels_secret(&txout, 1, Some(&secret)).unwrap(); - assert_eq!(dtx2.input_count, dtx.input_count); + assert_eq!(dtx2.input_count, 0, "txout meta omits n_in"); assert_eq!(douts2[0].script, outputs[0].script); - let dins = decode_inwit_secret(&inwit, dtx.input_count, Some(&secret)).unwrap(); + let dins = decode_inwit_secret(&inwit, tx.input_count, Some(&secret)).unwrap(); assert_eq!(dins[0].script_sig, inputs[0].script_sig); assert_eq!(dins[0].witness, inputs[0].witness); } @@ -2150,7 +2315,10 @@ fn packed_encode_decode_flags_and_error_arms() { assert_eq!(m2.txid, [0u8; 32]); let mut inwit = Vec::new(); encode_inwit_with_secret(&inputs, &mut inwit, None); - assert_eq!(scan_inwit_prevouts(&inwit, m.input_count).unwrap().len(), 2); + assert_eq!( + scan_inwit_prevouts(&inwit, tx.input_count).unwrap().len(), + 2 + ); let (m4, outs_rels, rels) = decode_packed_tx_outs_with_spender_rels(&raw, 2).unwrap(); assert_eq!(m4.txid, [0u8; 32]); assert_eq!(outs_rels.len(), 2); @@ -3240,20 +3408,20 @@ fn rec_meta(version: i32, locktime: u32, n_in: u32, n_out: u32) -> TxRecord { } #[test] -fn body_meta_v17_v1_locktime_zero_is_three_bytes() { +fn body_meta_v17_v1_locktime_zero_is_one_byte() { let rec = rec_meta(1, 0, 1, 1); let mut buf = Vec::new(); encode_body_meta_v17(&rec, &mut buf); assert_eq!( buf, - vec![0x89, 0x01], - "LAYOUT17|VER_1|LOCKTIME_ZERO + uleb input_count 1" + vec![0x89 | BODY_META_V17_N_IN_TXSTAT], + "LAYOUT17|VER_1|LOCKTIME_ZERO|N_IN_TXSTAT" ); let (got, n) = decode_body_meta_v17(&buf).unwrap(); - assert_eq!(n, 2); + assert_eq!(n, 1); assert_eq!(got.version, 1); assert_eq!(got.locktime, 0); - assert_eq!(got.input_count, 1); + assert_eq!(got.input_count, 0); assert_eq!(got.output_count, 0); } @@ -3262,7 +3430,11 @@ fn body_meta_v17_v2_locktime_zero() { let rec = rec_meta(2, 0, 1, 2); let mut buf = Vec::new(); encode_body_meta_v17(&rec, &mut buf); - assert_eq!(buf[0], 0x8A, "LAYOUT17|VER_2|LOCKTIME_ZERO"); + assert_eq!( + buf[0], + 0x8A | BODY_META_V17_N_IN_TXSTAT, + "LAYOUT17|VER_2|LOCKTIME_ZERO|N_IN_TXSTAT" + ); let (got, n) = decode_body_meta_v17(&buf).unwrap(); assert_eq!(n, buf.len()); assert_eq!(got.version, 2); @@ -3308,6 +3480,40 @@ fn body_meta_v17_rejects_missing_layout_bit() { } } +#[test] +fn decode_body_meta_v17_omitted_n_in() { + let rec = rec_meta(1, 0, 7, 1); + let mut buf = Vec::new(); + encode_body_meta_v17(&rec, &mut buf); + assert_eq!(buf, vec![0x89 | BODY_META_V17_N_IN_TXSTAT]); + let (got, n) = decode_body_meta_v17(&buf).unwrap(); + assert_eq!(n, 1); + assert_eq!(got.version, 1); + assert_eq!(got.locktime, 0); + assert_eq!(got.input_count, 0); +} + +#[test] +fn decode_body_meta_v17_reserved_bits_still_corrupt() { + let rec = rec_meta(1, 0, 1, 1); + let mut buf = Vec::new(); + encode_body_meta_v17(&rec, &mut buf); + buf[0] |= 0x20; + match decode_body_meta_v17(&buf) { + Err(StoreError::Corrupt(m)) => { + assert!(m.contains("reserved"), "{m}"); + } + other => panic!("expected reserved Corrupt, got {other:?}"), + } + buf[0] = (buf[0] & !0x20) | 0x40; + match decode_body_meta_v17(&buf) { + Err(StoreError::Corrupt(m)) => { + assert!(m.contains("reserved"), "{m}"); + } + other => panic!("expected reserved Corrupt, got {other:?}"), + } +} + fn p2pkh_script(h160: [u8; 20]) -> Vec { let mut s = vec![0x76, 0xa9, 0x14]; s.extend_from_slice(&h160); diff --git a/crates/rbitcoin-store/src/txstat.rs b/crates/rbitcoin-store/src/txstat.rs new file mode 100644 index 000000000..08a13efc3 --- /dev/null +++ b/crates/rbitcoin-store/src/txstat.rs @@ -0,0 +1,666 @@ +//! Dense **create_fk-ordered** confirm-time econ (`txstat.body`). +//! +//! Layout (schema 25): +//! ```text +//! txstat.body offset 0..32 — TableFile 16 + 16 pad +//! offset 32+(fk-1)×8 — 8-byte cell +//! txstat.ovf append-only tails — remaining ULEB bytes when the +//! four-field stream exceeds 8 B +//! txstat.blk offset 32+(header_fk-1)×16 — off:u64, len:u32, n_ovf:u32 +//! ``` +//! +//! Cell payload is four canonical ULEBs: `n_in`, `fee_sat`, `base` (non-witness +//! size), `wit_extra` (`total_size − base`). `size = base + wit_extra`, +//! `weight = 4×base + wit_extra`. All-zero cell = unstamped. A truncated ULEB +//! or fewer than four fields means the rest of the stream is in that header's +//! overflow blob (`encoded[8..]`). Pin / SH / tweaks do not open these files. + +use crate::error::StoreError; +use crate::file::{GrowPolicy, TableFile, FILE_HEADER_LEN}; +use rbitcoin_primitives::{read_uleb128, uleb128_len, write_uleb128, Fk, TableKind}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; + +/// Bytes before first body cell (TableFile header 16 + pad to 32). +pub const TXSTAT_BODY_HEADER: u64 = 32; +pub const TXSTAT_ENTRY_LEN: u64 = 8; +const BLK_HEADER: u64 = 32; +const BLK_SLOT: u64 = 16; + +/// Packed confirm-time econ. All-zero on disk is unstamped (not this struct). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct TxStatRow { + pub n_in: u32, + pub fee_sat: u64, + pub base: u32, + pub wit_extra: u32, +} + +impl TxStatRow { + pub fn size(&self) -> u64 { + u64::from(self.base).saturating_add(u64::from(self.wit_extra)) + } + + pub fn weight(&self) -> u64 { + u64::from(self.base) + .saturating_mul(4) + .saturating_add(u64::from(self.wit_extra)) + } + + pub fn has_witness(&self) -> bool { + self.wit_extra != 0 + } +} + +pub fn encode_stream(row: &TxStatRow) -> Result, StoreError> { + if row.n_in > u32::from(u16::MAX) { + return Err(StoreError::Corrupt("txstat n_in exceeds 16 bits")); + } + let mut v = Vec::new(); + write_uleb128(&mut v, u64::from(row.n_in)); + write_uleb128(&mut v, row.fee_sat); + write_uleb128(&mut v, u64::from(row.base)); + write_uleb128(&mut v, u64::from(row.wit_extra)); + Ok(v) +} + +pub fn pack_cell(row: &TxStatRow) -> Result<([u8; 8], Option>), StoreError> { + let s = encode_stream(row)?; + let mut cell = [0u8; 8]; + if s.len() <= 8 { + cell[..s.len()].copy_from_slice(&s); + Ok((cell, None)) + } else { + cell.copy_from_slice(&s[..8]); + Ok((cell, Some(s[8..].to_vec()))) + } +} + +fn read_canonical_uleb(buf: &[u8]) -> Result<(u64, usize), StoreError> { + let (v, n) = read_uleb128(buf).map_err(|e| StoreError::Corrupt(e.0))?; + if n != uleb128_len(v) { + return Err(StoreError::Corrupt("txstat uleb overlong")); + } + Ok((v, n)) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellParse { + Unstamped, + Complete(TxStatRow), + NeedTail, +} + +pub fn parse_cell(cell: [u8; 8]) -> Result { + if cell == [0u8; 8] { + return Ok(CellParse::Unstamped); + } + match parse_stream(&cell, true)? { + StreamParse::Complete(row) => Ok(CellParse::Complete(row)), + StreamParse::NeedMore => Ok(CellParse::NeedTail), + } +} + +enum StreamParse { + Complete(TxStatRow), + NeedMore, +} + +fn parse_stream(buf: &[u8], allow_trunc: bool) -> Result { + let mut off = 0usize; + let mut fields = [0u64; 4]; + for slot in &mut fields { + if off >= buf.len() { + if allow_trunc { + return Ok(StreamParse::NeedMore); + } + return Err(StoreError::Corrupt("txstat uleb truncated")); + } + match read_canonical_uleb(&buf[off..]) { + Ok((v, n)) => { + *slot = v; + off += n; + } + Err(StoreError::Corrupt("uleb128 truncated")) if allow_trunc => { + return Ok(StreamParse::NeedMore); + } + Err(e) => return Err(e), + } + } + if buf[off..].iter().any(|&b| b != 0) { + return Err(StoreError::Corrupt("txstat cell trailing non-zero")); + } + let n_in = u32::try_from(fields[0]).map_err(|_| StoreError::Corrupt("txstat n_in"))?; + let base = u32::try_from(fields[2]).map_err(|_| StoreError::Corrupt("txstat base"))?; + let wit_extra = + u32::try_from(fields[3]).map_err(|_| StoreError::Corrupt("txstat wit_extra"))?; + if n_in == 0 { + return Err(StoreError::Corrupt("txstat n_in zero in stamped cell")); + } + Ok(StreamParse::Complete(TxStatRow { + n_in, + fee_sat: fields[1], + base, + wit_extra, + })) +} + +pub fn parse_with_tail(cell: [u8; 8], tail: &[u8]) -> Result { + let mut buf = Vec::with_capacity(8 + tail.len()); + buf.extend_from_slice(&cell); + buf.extend_from_slice(tail); + match parse_stream(&buf, false)? { + StreamParse::Complete(row) => Ok(row), + StreamParse::NeedMore => Err(StoreError::Corrupt("invariant: txstat overflow missing")), + } +} + +pub fn n_in_from_cell(cell: [u8; 8]) -> Result, StoreError> { + if cell == [0u8; 8] { + return Ok(None); + } + let (n, _) = read_canonical_uleb(&cell)?; + if n == 0 { + return Ok(None); + } + let n_in = u32::try_from(n).map_err(|_| StoreError::Corrupt("txstat n_in"))?; + Ok(Some(n_in)) +} + +pub fn encode_ovf_blob(tails: &[(u16, Vec)]) -> Result, StoreError> { + let mut out = Vec::new(); + for (idx, rest) in tails { + if rest.is_empty() { + return Err(StoreError::Corrupt("txstat overflow empty tail")); + } + if rest.len() > 255 { + return Err(StoreError::Corrupt("txstat overflow tail")); + } + out.extend_from_slice(&idx.to_le_bytes()); + out.push(rest.len() as u8); + out.extend_from_slice(rest); + } + Ok(out) +} + +pub fn decode_ovf_blob(raw: &[u8]) -> Result)>, StoreError> { + let mut out = Vec::new(); + let mut off = 0usize; + while off < raw.len() { + if raw.len() - off < 3 { + return Err(StoreError::Corrupt("txstat overflow short")); + } + let idx = u16::from_le_bytes([raw[off], raw[off + 1]]); + let n = raw[off + 2] as usize; + off += 3; + if n == 0 || off + n > raw.len() { + return Err(StoreError::Corrupt("txstat overflow tail")); + } + out.push((idx, raw[off..off + n].to_vec())); + off += n; + } + Ok(out) +} + +fn tail_for_index(tails: &[(u16, Vec)], idx: u16) -> Result<&[u8], StoreError> { + match tails.binary_search_by_key(&idx, |(i, _)| *i) { + Ok(p) => Ok(tails[p].1.as_slice()), + Err(_) => Err(StoreError::Corrupt("invariant: txstat overflow missing")), + } +} + +/// Dense 8 B/create body plus per-header overflow tails. +pub struct TxStat { + body: TableFile, + ovf: TableFile, + blk: TableFile, + count: AtomicU64, + ovf_len: AtomicU64, +} + +impl TxStat { + pub fn create(dir: &Path) -> Result { + let body = TableFile::create(Self::body_path(dir), TableKind::TxStat)?; + let pad = vec![0u8; (TXSTAT_BODY_HEADER as usize).saturating_sub(FILE_HEADER_LEN)]; + if !pad.is_empty() { + body.write_at_pwrite(FILE_HEADER_LEN as u64, &pad)?; + } + let ovf = TableFile::create(Self::ovf_path(dir), TableKind::TxStatOvf)?; + ovf.set_grow_policy(GrowPolicy::Tight1MiB); + let blk = TableFile::create(Self::blk_path(dir), TableKind::TxStatBlk)?; + blk.set_grow_policy(GrowPolicy::Tight1MiB); + let bpad = vec![0u8; (BLK_HEADER as usize).saturating_sub(FILE_HEADER_LEN)]; + if !bpad.is_empty() { + blk.write_at_pwrite(FILE_HEADER_LEN as u64, &bpad)?; + } + Ok(Self { + body, + ovf, + blk, + count: AtomicU64::new(0), + ovf_len: AtomicU64::new(0), + }) + } + + pub fn open(dir: &Path) -> Result { + let leftover = dir.join("txfixed.body"); + if leftover.exists() { + std::fs::remove_file(&leftover).map_err(|e| StoreError::io(&leftover, e))?; + } + let body_path = Self::body_path(dir); + let body = if body_path.exists() { + TableFile::open(body_path, TableKind::TxStat)? + } else { + let b = TableFile::create(&body_path, TableKind::TxStat)?; + let pad = vec![0u8; (TXSTAT_BODY_HEADER as usize).saturating_sub(FILE_HEADER_LEN)]; + if !pad.is_empty() { + b.write_at_pwrite(FILE_HEADER_LEN as u64, &pad)?; + } + b + }; + let len = body.logical_len(); + let count = if len <= TXSTAT_BODY_HEADER { + 0 + } else { + (len - TXSTAT_BODY_HEADER) / TXSTAT_ENTRY_LEN + }; + let ovf_path = Self::ovf_path(dir); + let ovf = if ovf_path.exists() { + let f = TableFile::open(ovf_path, TableKind::TxStatOvf)?; + f.set_grow_policy(GrowPolicy::Tight1MiB); + f + } else { + let f = TableFile::create(&ovf_path, TableKind::TxStatOvf)?; + f.set_grow_policy(GrowPolicy::Tight1MiB); + f + }; + let ovf_len = ovf.data_len(); + let blk_path = Self::blk_path(dir); + let blk = if blk_path.exists() { + let f = TableFile::open(blk_path, TableKind::TxStatBlk)?; + f.set_grow_policy(GrowPolicy::Tight1MiB); + f + } else { + let f = TableFile::create(&blk_path, TableKind::TxStatBlk)?; + f.set_grow_policy(GrowPolicy::Tight1MiB); + let bpad = vec![0u8; (BLK_HEADER as usize).saturating_sub(FILE_HEADER_LEN)]; + if !bpad.is_empty() { + f.write_at_pwrite(FILE_HEADER_LEN as u64, &bpad)?; + } + f + }; + Ok(Self { + body, + ovf, + blk, + count: AtomicU64::new(count), + ovf_len: AtomicU64::new(ovf_len), + }) + } + + fn body_path(dir: &Path) -> PathBuf { + dir.join("txstat.body") + } + fn ovf_path(dir: &Path) -> PathBuf { + dir.join("txstat.ovf") + } + fn blk_path(dir: &Path) -> PathBuf { + dir.join("txstat.blk") + } + + pub fn count(&self) -> u64 { + self.count.load(Ordering::Acquire) + } + + pub fn truncate_to_count(&self, new_count: u64) -> Result<(), StoreError> { + let cur = self.count(); + if new_count > cur { + return Err(StoreError::Corrupt("txstat.body truncate past count")); + } + if new_count == cur { + return Ok(()); + } + let new_len = TXSTAT_BODY_HEADER + new_count * TXSTAT_ENTRY_LEN; + self.body.set_logical_len(new_len)?; + self.count.store(new_count, Ordering::Release); + Ok(()) + } + + pub fn extend_or_truncate_to(&self, new_count: u64) -> Result<(), StoreError> { + let cur = self.count(); + if new_count == cur { + return Ok(()); + } + if new_count < cur { + return self.truncate_to_count(new_count); + } + let new_len = TXSTAT_BODY_HEADER + new_count * TXSTAT_ENTRY_LEN; + self.body.set_logical_len(new_len)?; + self.count.store(new_count, Ordering::Release); + Ok(()) + } + + #[inline] + pub fn entry_offset(fk: u64) -> Result { + if fk == 0 { + return Err(StoreError::InvalidFk); + } + Ok(TXSTAT_BODY_HEADER + (fk - 1) * TXSTAT_ENTRY_LEN) + } + + pub fn get_cell(&self, fk: Fk) -> Result<[u8; 8], StoreError> { + let id = fk.get().ok_or(StoreError::InvalidFk)?; + if id > self.count() { + return Err(StoreError::NotFound); + } + let off = Self::entry_offset(id)?; + let mut buf = [0u8; 8]; + self.body.read_at(off, &mut buf)?; + Ok(buf) + } + + pub fn n_in_at(&self, fk: Fk) -> Result, StoreError> { + n_in_from_cell(self.get_cell(fk)?) + } + + pub fn get_row_merged( + &self, + fk: Fk, + first_fk: u64, + blob: &[u8], + ) -> Result, StoreError> { + let cell = self.get_cell(fk)?; + match parse_cell(cell)? { + CellParse::Unstamped => Ok(None), + CellParse::Complete(row) => Ok(Some(row)), + CellParse::NeedTail => { + let id = fk.get().ok_or(StoreError::InvalidFk)?; + if id < first_fk { + return Err(StoreError::Corrupt("invariant: txstat overflow missing")); + } + let idx = u16::try_from(id - first_fk) + .map_err(|_| StoreError::Corrupt("txstat overflow index"))?; + let tails = decode_ovf_blob(blob)?; + let rest = tail_for_index(&tails, idx)?; + Ok(Some(parse_with_tail(cell, rest)?)) + } + } + } + + pub fn get_range( + &self, + first: u64, + last: u64, + blob: Option<&[u8]>, + ) -> Result>, StoreError> { + if last < first { + return Ok(Vec::new()); + } + let n = self.count(); + if first == 0 || last > n { + return Err(StoreError::NotFound); + } + let count = (last - first + 1) as usize; + let off = Self::entry_offset(first)?; + let mut raw = vec![0u8; count * TXSTAT_ENTRY_LEN as usize]; + let rc = crate::bulk_io::pread_single(self.body.read_fd(), off, &mut raw); + if rc < 0 { + return Err(StoreError::io( + self.body.path(), + std::io::Error::from_raw_os_error(-rc), + )); + } + if (rc as usize) != raw.len() { + self.body.pread_at(off, &mut raw)?; + } + let tails = match blob { + Some(b) if !b.is_empty() => decode_ovf_blob(b)?, + _ => Vec::new(), + }; + let mut out = Vec::with_capacity(count); + for i in 0..count { + let s = i * TXSTAT_ENTRY_LEN as usize; + let cell: [u8; 8] = raw[s..s + 8].try_into().unwrap(); + match parse_cell(cell)? { + CellParse::Unstamped => out.push(None), + CellParse::Complete(row) => out.push(Some(row)), + CellParse::NeedTail => { + let idx = u16::try_from(i) + .map_err(|_| StoreError::Corrupt("txstat overflow index"))?; + let rest = tail_for_index(&tails, idx)?; + out.push(Some(parse_with_tail(cell, rest)?)); + } + } + } + Ok(out) + } + + pub fn append_batch( + &self, + base_count: u64, + rows: &[TxStatRow], + ) -> Result)>, StoreError> { + if rows.is_empty() { + return Ok(Vec::new()); + } + let cur = self.count.load(Ordering::Acquire); + if cur != base_count { + return Err(StoreError::Corrupt("txstat.body count mismatch on append")); + } + let start = TXSTAT_BODY_HEADER + base_count * TXSTAT_ENTRY_LEN; + let mut blob = Vec::with_capacity(rows.len() * TXSTAT_ENTRY_LEN as usize); + let mut tails = Vec::new(); + for (i, row) in rows.iter().enumerate() { + let (cell, tail) = pack_cell(row)?; + blob.extend_from_slice(&cell); + if let Some(t) = tail { + tails.push((base_count + 1 + i as u64, t)); + } + } + self.body.write_at_pwrite(start, &blob)?; + let new = base_count + rows.len() as u64; + self.count.store(new, Ordering::Release); + Ok(tails) + } + + pub fn write_row(&self, fk: Fk, row: &TxStatRow) -> Result<(), StoreError> { + let id = fk.get().ok_or(StoreError::InvalidFk)?; + if id > self.count() { + return Err(StoreError::NotFound); + } + let (cell, tail) = pack_cell(row)?; + if tail.is_some() { + return Err(StoreError::Corrupt("txstat overflow needs header blob")); + } + let off = Self::entry_offset(id)?; + self.body.write_at_pwrite(off, &cell)?; + Ok(()) + } + + pub fn write_block_rows( + &self, + header_fk: Fk, + first_fk: u64, + rows: &[TxStatRow], + ) -> Result<(), StoreError> { + if rows.is_empty() { + return self.put_header_blob(header_fk, &[]); + } + let last = first_fk + .checked_add(rows.len() as u64 - 1) + .ok_or(StoreError::Corrupt("txstat block last fk"))?; + if last > self.count() { + return Err(StoreError::NotFound); + } + let start = Self::entry_offset(first_fk)?; + let mut blob = Vec::with_capacity(rows.len() * 8); + let mut tails = Vec::new(); + for (i, row) in rows.iter().enumerate() { + let (cell, tail) = pack_cell(row)?; + blob.extend_from_slice(&cell); + if let Some(t) = tail { + let idx = + u16::try_from(i).map_err(|_| StoreError::Corrupt("txstat overflow index"))?; + tails.push((idx, t)); + } + } + self.body.write_at_pwrite(start, &blob)?; + let raw = encode_ovf_blob(&tails)?; + self.put_header_blob(header_fk, &raw) + } + + pub fn put_overflows_for_headers( + &self, + ranges: &[(Fk, Fk, u32)], + tails: &[(u64, Vec)], + ) -> Result<(), StoreError> { + if ranges.is_empty() { + if !tails.is_empty() { + return Err(StoreError::Corrupt("txstat overflow without header")); + } + return Ok(()); + } + for &(hfk, first, n) in ranges { + let Some(first_id) = first.get() else { + continue; + }; + let last = first_id.saturating_add(u64::from(n.saturating_sub(1))); + let mut chunk = Vec::new(); + for (fk, rest) in tails { + if *fk < first_id || *fk > last { + continue; + } + let idx = u16::try_from(*fk - first_id) + .map_err(|_| StoreError::Corrupt("txstat overflow index"))?; + chunk.push((idx, rest.clone())); + } + if chunk.is_empty() { + continue; + } + let raw = encode_ovf_blob(&chunk)?; + self.put_header_blob(hfk, &raw)?; + } + Ok(()) + } + + pub fn put_header_blob(&self, header_fk: Fk, blob: &[u8]) -> Result<(), StoreError> { + let hfk = header_fk.get().ok_or(StoreError::InvalidFk)?; + let (off, len, n) = if blob.is_empty() { + (0u64, 0u32, 0u32) + } else { + let at = FILE_HEADER_LEN as u64 + self.ovf_len.load(Ordering::Acquire); + self.ovf.write_at_pwrite(at, blob)?; + self.ovf_len.store(self.ovf.data_len(), Ordering::Release); + let nlen = blob.len() as u64; + let n_ovf = decode_ovf_blob(blob)?.len() as u32; + ( + at, + u32::try_from(nlen).map_err(|_| StoreError::Corrupt("txstat ovf len"))?, + n_ovf, + ) + }; + let slot_off = BLK_HEADER + (hfk - 1) * BLK_SLOT; + let mut slot = [0u8; 16]; + slot[0..8].copy_from_slice(&off.to_le_bytes()); + slot[8..12].copy_from_slice(&len.to_le_bytes()); + slot[12..16].copy_from_slice(&n.to_le_bytes()); + self.blk.write_at_pwrite(slot_off, &slot)?; + Ok(()) + } + + pub fn header_blob(&self, header_fk: Fk) -> Result, StoreError> { + let hfk = header_fk.get().ok_or(StoreError::InvalidFk)?; + let slot_off = BLK_HEADER + (hfk - 1) * BLK_SLOT; + if slot_off + BLK_SLOT > self.blk.logical_len() { + return Ok(Vec::new()); + } + let mut slot = [0u8; 16]; + self.blk.read_at(slot_off, &mut slot)?; + let off = u64::from_le_bytes(slot[0..8].try_into().unwrap()); + let len = u32::from_le_bytes(slot[8..12].try_into().unwrap()) as usize; + if len == 0 { + return Ok(Vec::new()); + } + let mut buf = vec![0u8; len]; + self.ovf.read_at(off, &mut buf)?; + Ok(buf) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::testutil::TempDir; + + fn tiny(n_in: u32, fee: u64, base: u32, wit: u32) -> TxStatRow { + TxStatRow { + n_in, + fee_sat: fee, + base, + wit_extra: wit, + } + } + + #[test] + fn create_empty_then_extend_zeros() { + let dir = TempDir::labeled("txstat-extend").unwrap(); + let t = TxStat::create(&dir).unwrap(); + assert_eq!(t.count(), 0); + t.extend_or_truncate_to(2).unwrap(); + assert_eq!(t.count(), 2); + assert_eq!(t.get_cell(Fk(1)).unwrap(), [0u8; 8]); + assert_eq!(t.get_cell(Fk(2)).unwrap(), [0u8; 8]); + t.truncate_to_count(1).unwrap(); + assert_eq!(t.count(), 1); + assert!(t.get_cell(Fk(2)).is_err()); + } + + #[test] + fn uleb_typical_fits_in_cell() { + let row = tiny(1, 1000, 110, 112); + let (cell, tail) = pack_cell(&row).unwrap(); + assert!(tail.is_none()); + assert_eq!(parse_cell(cell).unwrap(), CellParse::Complete(row)); + assert_eq!(n_in_from_cell(cell).unwrap(), Some(1)); + assert_eq!(row.size(), 222); + assert_eq!(row.weight(), 4 * 110 + 112); + } + + #[test] + fn overflow_is_remaining_bytes_only() { + let row = tiny(1, 50_000, 200, 200_000); + let stream = encode_stream(&row).unwrap(); + assert!(stream.len() > 8, "fat wit must miss 8 B"); + let (cell, tail) = pack_cell(&row).unwrap(); + let tail = tail.expect("tail"); + assert_eq!(tail.as_slice(), &stream[8..]); + assert_eq!(parse_cell(cell).unwrap(), CellParse::NeedTail); + assert_eq!(n_in_from_cell(cell).unwrap(), Some(1)); + assert_eq!(parse_with_tail(cell, &tail).unwrap(), row); + } + + #[test] + fn header_blob_roundtrip() { + let dir = TempDir::labeled("txstat-ovf").unwrap(); + let t = TxStat::create(&dir).unwrap(); + let small = tiny(1, 1000, 110, 0); + let fat = tiny(2, 2_000_000, 400, 200_000); + t.append_batch(0, &[small, fat]).unwrap(); + t.write_block_rows(Fk(1), 1, &[small, fat]).unwrap(); + let blob = t.header_blob(Fk(1)).unwrap(); + assert!(!blob.is_empty()); + let got = t.get_range(1, 2, Some(&blob)).unwrap(); + assert_eq!(got[0], Some(small)); + assert_eq!(got[1], Some(fat)); + assert_eq!(t.get_row_merged(Fk(2), 1, &blob).unwrap(), Some(fat)); + } + + #[test] + fn leftover_txfixed_body_unlinked_on_open() { + let dir = TempDir::labeled("txstat-leftover").unwrap(); + std::fs::write(dir.join("txfixed.body"), b"junk").unwrap(); + let _t = TxStat::open(&dir).unwrap(); + assert!(!dir.join("txfixed.body").exists()); + assert!(dir.join("txstat.body").is_file()); + } +} diff --git a/docs/invariants.md b/docs/invariants.md index 06cb79449..fe2bf5709 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -17,7 +17,7 @@ with `Err(…Corrupt("invariant: …"))` (and `debug_assert!` where useful). Do | **Load miss** | Spend annotate without `spent_range` abs; body decode without idx range; pin without outs for need_vouts; ensure without abs for a spend edge | Assert / hard Err; fix lookup/load | | **Environment** | bulk IO backend uring vs pread/pwrite (single backend trait) | Keep modality only | | **Protocol** | BIP30 multi-spender confirmed-strong walk; same-block spends; coinbase null create | Real branches (not soft recovery) | -| **Format migrate** | Occupied 22 rewrites `create.loc.ovf` 12 B→16 B; occupied 23 rewrites `header.body` 88 B→96 B; `meta` to live `SCHEMA_VERSION`. Leftover fuse8 v1 / flat idx / Shared SH body / Paged pack8 **refuse** | Soft migrate (tmp+rename) or explicit wipe/rebuild | +| **Format migrate** | Occupied 22 rewrites `create.loc.ovf` 12 B→16 B; occupied 23 rewrites `header.body` 88 B→96 B; occupied 24 zero-extends `txstat.body`; `meta` to live `SCHEMA_VERSION`. Leftover fuse8 v1 / flat idx / Shared SH body / Paged pack8 **refuse** | Soft migrate (tmp+rename) or explicit wipe/rebuild | | **API / product** | RPC body from store; Electrum mempool after chain; compact → getdata | Keep | **One path.** Do not reintroduce a silent colder alternate when lookup or load diff --git a/docs/personal-node-plans/09-inwit-prune.md b/docs/personal-node-plans/09-inwit-prune.md index 87c846dd5..bf3a86959 100644 --- a/docs/personal-node-plans/09-inwit-prune.md +++ b/docs/personal-node-plans/09-inwit-prune.md @@ -203,7 +203,7 @@ Heights **above** the watermark behave as today. Below: table. | `getblock` verbosity **1** | **Keep** (`block_txids` / `txid.body`) | txid list, no vin | | `getblock` verbosity **2** | `-8` pruned | Full txs need inwit | | `getblockheader` / `getblockhash` / `getblockcount` | Keep | Headers only | -| `getblockstats` | `-8` / existing `block body not in store` | Reconstruct | +| `getblockstats` | serve from `txstat` when stamped, else `-8` | Reconstruct + lazy-stamp if unstamped and inwit remains | | `getrawtransaction` | **`-8` `Transaction not available (pruned data)`** | Not `-5` unknown txid | | `decoderawtransaction` | n/a (client hex) | | | `gettxout` | **Keep** | Class A outs + spent | @@ -230,13 +230,14 @@ Heights **above** the watermark behave as today. Below: table. `GET /broadcast`, `POST /txs/test`, tx JSON `sigops`, unix listen). **Partial tx object** (when inwit is gone): `txid`, `version`, `locktime`, -`vout[]` from txout, `status`, `"pruned": true`. **Omit** `vin`, `fee`, -`size`, `weight`, `sigops`. Same shape on list rows so `/txs` paging matches -`/txs/summary` (do not drop the txid from the page). +`vout[]` from txout, `status`, `"pruned": true`. **Omit** `vin`. Fill `fee` / +`size` / `weight` from stamped `txstat` when those rows exist; +unstamped pruned omits those keys too. Same shape on list rows so `/txs` +paging matches `/txs/summary` (do not drop the txid from the page). | Route | Pruned confirmed tx/block | Why | |-------|---------------------------|-----| -| `GET /tx/:txid` JSON | **200** partial object | vout+status still true; explorer UIs that require `vin` fail closed on missing key (better than a fake coinbase) | +| `GET /tx/:txid` JSON | **200** partial object; `fee`/`size`/`weight` from `txstat` when stamped | vout+status still true; still omit `vin` | | `GET /tx/:txid/hex`, `/raw` | **404** body `pruned` | Wire | | `GET /tx/:txid/status` | **Keep** (200) | Header + fk, no inwit | | `GET /tx/:txid/merkle-proof` | **Keep** | txids | @@ -245,7 +246,7 @@ Heights **above** the watermark behave as today. Below: table. | `GET /block/:hash` JSON | **200**; omit `size`/`weight` (or only if we cannot compute them); `"pruned": true` | Header + txids; not a witness size | | `GET /block/:hash/raw` | **404** | Full witness block | | `GET /block/:hash/header` `/status` `/txids` `/txid/:i` | **Keep** | header + `txid.body` | -| `GET /block/:hash/txs` (public 25/page and **632** `GET /internal/block/:hash/txs`) | **200** pages of **partial** tx objects | Same omit-vin rule; do not 404 the whole block list | +| `GET /block/:hash/txs` (public 25/page and **632** `GET /internal/block/:hash/txs`) | **200** pages of **partial** tx objects | Same omit-vin rule; stamped `txstat` fills fee/size/weight | | Address `/` stats, `/utxo`, `/txs/summary` | **Keep** | SH + values from txout | | Address `/txs`, `/txs/chain`, **632** `POST /addresses/txs` | **200** with partial rows for pruned txs | Keep paging aligned with summary | | **632** `?after_txid=` | Keep (txid cursor) | | diff --git a/docs/road-to-1.0.md b/docs/road-to-1.0.md index 4ba136275..45314a2cd 100644 --- a/docs/road-to-1.0.md +++ b/docs/road-to-1.0.md @@ -5,7 +5,7 @@ Day-to-day ranked work stays in [`quality.md`](./quality.md). **Today (0.7.99):** in-tree toward **0.8.0**. Last published tag is **0.7.0** (`v0.7.x` patch line). Current bytes and refuse: -[`SCHEMA.md`](../SCHEMA.md) (`SCHEMA_VERSION = 24`). Occupied 0.6.x +[`SCHEMA.md`](../SCHEMA.md) (`SCHEMA_VERSION = 25`). Occupied 0.6.x (schema 20) Class A refuses (wipe + IBD). Electrum/Esplora need `--sh-index` (default off). BIP324 v2-only. Install is a GitHub Release (Linux musl; Windows/Darwin snapshots). Nightly diff --git a/docs/rpc.md b/docs/rpc.md index 334401778..11157b840 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -85,7 +85,7 @@ still wait for durable SH when shindex is on. | `echo` | Testing RPC. Returns arguments as a positional array. AuthServiceProxy `{args: [...], argN: ...}` is peeled only here. Mixed `submitpackage`/`sendrawtransaction`/`testmempoolaccept` `{args, maxfeerate}` is expanded in the Core-functional proxy, not on the node. | | `getblockchaininfo` / `getblockcount` / `getbestblockhash` / `getblockhash` | Chain tip. `getblockcount` / `getbestblockhash` wait for the in-flight tip-accept job (not the rest of a catch-up burst). `headers` is the best known header height (`submitheader` / P2P headers may lead `blocks`). `chainwork` is summed header work (regtest 2 per block). `size_on_disk` is a walk of `{datadir}/store` file lengths (plus `--datadir-cold` inwit when split). `verificationprogress` is `blocks / headers` clamped to `[0, 1]` (`1.0` when `headers` is 0). `initialblockdownload` is the Core RPC name for **relay-inhibited**: `--min-chain-work` and `--max-tip-age` after densify/`enter_tip_mode`, not “still catching up”. `--prune-inwit` sets `pruned: true` and `pruneheight` (omitted when off). | | `getblockheader` / `getblock` (verbosity 0/1/2) | Archive reconstruct. `getblockheader` includes `chainwork`. Verbosity 0/2 below `pruneheight` is `-8` `Block not available (pruned data)`; verbosity 1 (txids) stays. | -| `getblockstats` | All networks. Reconstruct the block; Core named keys `hash_or_height` / `stats`. Fees from archive prevouts. Genesis excluded from actual UTXO counts. OP_RETURN unspendable. Reconstruct miss is `block body not in store`. Dummy `blk00000.dat` is shim-only so `rpc_getblockstats.py`'s rename-file needle stays Core-phrased. Pruned height is `-8` Core pruned text. | +| `getblockstats` | All networks. Stamped `txstat` fee/size/weight/`n_in`; `utxo_increase` is `outs − ins`. Omits Core coins-DB `utxo_size_inc` / `utxo_size_inc_actual` / `utxo_increase_actual` (selecting those names is invalid statistic `-8`). Unstamped leftover reconstructs then lazy-stamps. Reconstruct miss is `block body not in store`. Dummy `blk00000.dat` is shim-only so `rpc_getblockstats.py`'s rename-file needle stays Core-phrased. Pruned unstamped height is `-8` Core pruned text. | | `getdifficulty` | From tip bits | | `getnetworkinfo` / `getconnectioncount` / `getpeerinfo` | BIP324 v2-only; `getpeerinfo` is the live session table. `timeoffset` is VERSION clock minus connect time (`0` before handshake). `synced_headers` is the height of that peer's advertised best block when we know it, else `-1`. `synced_blocks` is that height when the hash is on our best chain, else `-1`. `getnetworkinfo.timeoffset` is the median of outbound handshake-complete offsets (`0` if none). `mapped_as` is present when `--asmap` / `{datadir}/ip_asn.dat` mapped the peer (Core field; omitted without a map or ASN 0). `version` is rbitcoin semver as a Core integer (`major*10000+minor*100+patch`: `0.1.0` → `100`, `0.5.0` → `500`, `0.6.0` → `600`, `0.6.99` → `699`, `0.7.0` → `700`, `0.7.99` → `799`), not a Core release. `localservices` matches advertised `NETWORK\|WITNESS\|P2P_V2`, or `NETWORK_LIMITED\|WITNESS\|P2P_V2` under `--prune-inwit`. `localaddresses` lists `--external-ip` (`score` = Core `LOCAL_MANUAL`) | | `getnettotals` | All networks. Raw TCP `totalbytesrecv` / `totalbytessent` on live sessions. `uploadtarget` is a Core-shaped stub (`target` 0). |