diff --git a/COMPAT.md b/COMPAT.md index 09fdcba6c..dd0d6891a 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -226,7 +226,7 @@ App `ServeLimits` always on (same model as Electrum). | Block | done | `/block/:hash` JSON, `/raw`, `/status`, `/header`, `/txids`, `/txid/:i`, `/txs[/:start]`. JSON `bits` is the compact-target **u32** (Esplora schema, not Core hex). `size` / `weight` are BIP144 total size and BIP141 weight (witness included). | | Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` (`vin` from the spent slot; unspent omits it). Below seqsigwit prune: JSON is 200 partial (`pruned: true`, vout+status, empty `vin`); stamped `txstat` fills `fee`/`size`/`weight`. Mempool-only txs (not in Class A) use the wire body from the mempool hub (`vin`/`vout`/`size`/`weight`/`fee`/`sigops`, `status.confirmed` false) including `GET /tx/:txid/status`. Live `/outspend(s)` overlay mempool spends of confirmed coins; `?asof=` omits mempool. `?asof=` on `/status` and `/outspend(s)`: confirmed/spent as of that ancestor; 404 if not on chain. | | Address / scripthash | done | stats + `/utxo` + `/txs` + `/txs/mempool` + `/txs/chain[/:last_seen_txid]` + `/txs/summary[/:last_seen_txid]` (dialect; next row). `/utxo` matches Electrum listunspent (mempool funding + drop mempool-spent confirmed); `/txs` and `/txs/mempool` use full Esplora tx JSON for mempool-only rows (wire from the hub). `?after_txid=` on `/txs` and `/txs/summary` skips through that tx (mempool then chain); unknown or unparseable → **422** `after_txid not found`. `POST /addresses/txs` and `POST /scripthashes/txs` (and `/txs/summary`) take a JSON array (max **300**; over → **422** `body too long`); unique scripts join once per request, merge newest-first. Unix/loopback `X-Rbitcoin-Client`: last-1 GET join + last-bulk POST (16 MiB packed/client), 30s idle, 256 clients, singleflight on the same `(client, sh)`. Public TCP ignores the header (stack-local join). Concurrent different GET scripts replace last-1. Unbounded process LRU stays **X-M3**. Needs SH finalize. Stamp is visible SH (durable + pending write-behind), matching live tip while jobs sit in RAM. `?asof=` on `/`, `/utxo`, `/txs`, `/txs/chain`, `/txs/summary`: confirmed join at that ancestor **at or behind visible SH**, **no** mempool; headers are the asof hash; 404 if not on chain or ahead of visible SH. | -| `/txs/summary` | dialect | **Not** in Blockstream Esplora [`API.md`](https://github.com/Blockstream/esplora/blob/master/API.md). Compact `{txid, value, height, time}` like mempool.space `/address/:addr/txs/summary`. Confirmed only (25/page, newest first); path cursor `/:last_seen_txid` and mempool.space `?after_txid=` (unknown → **422**). `value` is net sats for that script in that tx (funded − spent). `time` is the confirming header timestamp (`0` if the header is missing). Mempool rows stay on `/txs` and `/txs/mempool`. Over `--max-sh-creates` → **503**. | +| `/txs/summary` | dialect | **Not** in Blockstream Esplora [`API.md`](https://github.com/Blockstream/esplora/blob/master/API.md). Compact `{txid, value, height, time, tx_position}` like mempool.space `/address/:addr/txs/summary`. Confirmed only (25/page, newest first); path cursor `/:last_seen_txid` and mempool.space `?after_txid=` (unknown → **422**). `value` is net sats for that script in that tx (funded − spent). `time` is the confirming header timestamp (`0` if the header is missing). `tx_position` is that tx's index in the block. Mempool rows stay on `/txs` and `/txs/mempool`. A page is served above `--max-sh-creates`; unpaged stats and `/utxo` still **503**. | | Mempool / fees | done | `/mempool`, `/mempool/txids`, `/mempool/txids/page[/:last]`, `/mempool/recent` (accept-order ring), `/fee-estimates` (sat/vB at the estimator's 1 sat/kvB resolution; a target without an estimate is left out, and **503** `fee estimates unavailable` when none has one). mempool.space `/api/v1/fees/recommended` is its backend's surface, not served here (**404**) | | `POST /tx` | done | broadcast via mempool hub; **503** if hub absent. `GET /broadcast?tx=` is the same admit path (legacy electrs). | | `POST /txs/test` | done | JSON array of hex (max 25) → `MempoolHub::test_accept` (no admit). `?maxfeerate=` is BTC/kvB like electrs (`0` unlimited; omitted → 0.1 BTC/kvB). | diff --git a/TESTING.md b/TESTING.md index b7e10ed50..2b465c212 100644 --- a/TESTING.md +++ b/TESTING.md @@ -342,7 +342,7 @@ Prefer **one high-level scenario** per behavior cluster. Delete lower-level test | `electrum_tweaks_subscribe_streams_then_done` | Electrum | Cake `tweaks.subscribe`: one-height result, per-height notifies, `done`, and BIP352 hash-bind on a P2WPKH→P2TR spend. Keep zero-chunk / pre-taproot units | | `electrum_max_connections_rejects_extra_client` | Electrum | TCP cap drops the extra client | | `electrum_idle_timeout_disconnects_quiet_client` | Electrum | Idle timeout closes a quiet socket | -| `esplora_broadcast_visible_in_rpc_and_electrum` | Node + Electrum + Esplora + RPC | One `run_p2p` datadir: HTTP `sendrawtransaction` / `testmempoolaccept` (allowed, missing-or-spent, exact 100 sat/kvB min-relay accept + one-sat-under reject, RBF one-sat-short incremental reject + exact incremental accept); Esplora `POST /tx` parent and mempool child appear in `getrawmempool` and Electrum mempool/history (`fee` on unconfirmed, including child `height = -1`); Electrum `listunspent` of that child is `height=-1` and the parent UTXO drops; process `gettxout` / `getchaintips`; Esplora `POST /txs/package` 1p1c (including parent-alone below min-relay + paying child), 25-tx accept, 26-tx and over-weight `package too large`; serving-only `submitpackage` refuses (relay off); live `GET /mempool` / `/mempool/txids` / `/mempool/recent` / `/fee-estimates` answers 503 while flow is cold and the chain holds too little fee history (a thin live pool alone sets no rate); process `getmempoolancestors` / descendants / cluster / `gettxspendingprevout` / feerate diagram / verbose `getrawmempool` on that 1p1c; `waitforblockheight` timeout=0 while behind returns the live tip; GBT stale `longpollid` is immediate; current id / `waitfornewblock` / `waitforblockheight` wake on the pad `generate`; `getblockhash` tip ok / tip+1 `-8`; unknown `getblock` `-5`; verbosity 0 hex and 2 vin/vout; `GET /blocks` 10 newest, `/blocks/0` and `/blocks/:tip` (start past tip clamps); `/block/:hash/txs/:start` last page shorter than 25, one-past last page `[]` (not 404), unknown hash 404; `/block/:hash/txids` + coinbase merkle-proof + unspent `outspend/0`; `/tx/:id/outspends`; `/block` JSON/raw/status/`txid/0` (OOB 404); `/tx/:id/raw` vs hex; merkleblock-proof; `/block-height` (missing 404); `/block/:hash/header` 160 hex; `/tx/:id/status` + full JSON (`unknown` OP_TRUE type, coinbase vin) and missing-tx 404s; OP_TRUE scripthash info/summary/utxo/`txs/chain` cursor and combined `/txs`. Keep crate no-hub mempool/fees/POST 503, reconstruct meters, header wire match, and `tx_status_json`; Esplora `/tx/:id/status` confirms the package parent on generate; `generate` includes those txs (parent before child) then leaves IBD (relay on); `scantxoutset` drops the spent coinbase and still sees a non-coinbase unspent; `submitpackage` maxfeerate reject, 1p1c success, already-in-mempool continue, below-min-relay parent + paying child success, 26-tx / over-weight `package too large`; immature coinbase sendraw rejects. Keep `accept.rs` reject units, package JSON errors, RPC dry-run orphan-count, leftover `gettxout` include_mempool / disconnected / leftover, `generate_selects_chained_mempool_parent_first`, `submitpackage_child_fail_keeps_parent`, maxburn `submitpackage`, and wait-on-stop units. Unix `--rpc-socket` (mode 0660, no datadir `rpc.sock`) `getblockcount` without Authorization; TCP `GET`/`POST /internal/mempool/txs`; `GET /internal/block/:hash/txs` full list vs public 25/page; `POST /internal/txs/outspends/by-txid` same-length unknown `[]` slot; `GET /address-prefix/bc1` **404**; unauthenticated Core REST on the RPC listener (`chaininfo`, block hash/headers/block/tx, mempool info/contents, `getutxos`, `deploymentinfo`, basic `blockfilter` bin/hex/json) and `getblockfilter` with `--block-filter-index` | +| `esplora_broadcast_visible_in_rpc_and_electrum` | Node + Electrum + Esplora + RPC | One `run_p2p` datadir: HTTP `sendrawtransaction` / `testmempoolaccept` (allowed, missing-or-spent, exact 100 sat/kvB min-relay accept + one-sat-under reject, RBF one-sat-short incremental reject + exact incremental accept); Esplora `POST /tx` parent and mempool child appear in `getrawmempool` and Electrum mempool/history (`fee` on unconfirmed, including child `height = -1`); Electrum `listunspent` of that child is `height=-1` and the parent UTXO drops; process `gettxout` / `getchaintips`; Esplora `POST /txs/package` 1p1c (including parent-alone below min-relay + paying child), 25-tx accept, 26-tx and over-weight `package too large`; serving-only `submitpackage` refuses (relay off); live `GET /mempool` / `/mempool/txids` / `/mempool/recent` / `/fee-estimates` answers 503 while flow is cold and the chain holds too little fee history (a thin live pool alone sets no rate); process `getmempoolancestors` / descendants / cluster / `gettxspendingprevout` / feerate diagram / verbose `getrawmempool` on that 1p1c; `waitforblockheight` timeout=0 while behind returns the live tip; GBT stale `longpollid` is immediate; current id / `waitfornewblock` / `waitforblockheight` wake on the pad `generate`; `getblockhash` tip ok / tip+1 `-8`; unknown `getblock` `-5`; verbosity 0 hex and 2 vin/vout; `GET /blocks` 10 newest, `/blocks/0` and `/blocks/:tip` (start past tip clamps); `/block/:hash/txs/:start` last page shorter than 25, one-past last page 404, unknown hash 404; `/block/:hash/txids` + coinbase merkle-proof + unspent `outspend/0`; `/tx/:id/outspends`; `/block` JSON/raw/status/`txid/0` (OOB 404); `/tx/:id/raw` vs hex; merkleblock-proof; `/block-height` (missing 404); `/block/:hash/header` 160 hex; `/tx/:id/status` + full JSON (`unknown` OP_TRUE type, coinbase vin) and missing-tx 404s; OP_TRUE scripthash info/summary/utxo/`txs/chain` cursor and combined `/txs`. Keep crate no-hub mempool/fees/POST 503, reconstruct meters, header wire match, and `tx_status_json`; Esplora `/tx/:id/status` confirms the package parent on generate; `generate` includes those txs (parent before child) then leaves IBD (relay on); `scantxoutset` drops the spent coinbase and still sees a non-coinbase unspent; `submitpackage` maxfeerate reject, 1p1c success, already-in-mempool continue, below-min-relay parent + paying child success, 26-tx / over-weight `package too large`; immature coinbase sendraw rejects. Keep `accept.rs` reject units, package JSON errors, RPC dry-run orphan-count, leftover `gettxout` include_mempool / disconnected / leftover, `generate_selects_chained_mempool_parent_first`, `submitpackage_child_fail_keeps_parent`, maxburn `submitpackage`, and wait-on-stop units. Unix `--rpc-socket` (mode 0660, no datadir `rpc.sock`) `getblockcount` without Authorization; TCP `GET`/`POST /internal/mempool/txs`; `GET /internal/block/:hash/txs` full list vs public 25/page; `POST /internal/txs/outspends/by-txid` same-length unknown `[]` slot; `GET /address-prefix/bc1` **404**; unauthenticated Core REST on the RPC listener (`chaininfo`, block hash/headers/block/tx, mempool info/contents, `getutxos`, `deploymentinfo`, basic `blockfilter` bin/hex/json) and `getblockfilter` with `--block-filter-index` | | `fee_history_backfills_from_the_chain_when_relay_starts` | Node + RPC | `run_p2p` on a mature regtest datadir, started twice; `generate` leaves IBD and turns relay on; the preload writes the fee history file and the restart extends it; with flow cold and too little history, `estimatesmartfee` answers Core's insufficient-data shape. Rates from a ready history are the hub's `far_horizon_follows_block_history_not_pool_tail`; the success object is `smart_fee_json` | | `node_listen_and_exit` | Node + Electrum + Esplora + RPC | One `run_p2p` datadir, restarted with its one `--connect` refusing (a pinned connect at genesis still enters tip mode): a taken `--health-listen` port stops the start before the store opens; a junk `peers` file and a missing `--asmap` start an empty book and exit, and the saved book records the refused connect; the next start loads that book and a valid `ip_asn.dat`, and Esplora, Electrum, and RPC answer at genesis until `stop`. On that start `/healthz` is 200 while genesis `/readyz` is initial block download and `/metrics` reports the same; `generate` makes `/readyz` 200 and the scrape ready at height 1. An Electrum port and an RPC port another process holds warn, `/readyz` names both, and `/metrics` is absent without `--metrics`. Without `--connect` and with seeds on, regtest resolves none and the node exits short of tip mode; after a `--prune-seqsigwit` start, an unpruned start refuses. Live peers are `node_run_p2p_short`. The `/readyz` phase, lag, and tip-age table stays `readiness_reports_the_first_failing_gate`. A timed-out probe keeping its permit, the full-gate 503, and the routes answering again are `health_gate_caps` | | `tor_control_onion_lifecycle` | Node + RPC | One `run_p2p` datadir against a fake Tor control port and SAM bridge (live Tor and i2pd are overlay-functional). A cookie from another Tor (SAFECOOKIE server hash mismatch), a 2-byte cookie, and a Tor that offers only plain COOKIE each refuse the start, and none sends `AUTHENTICATE`. Password auth with `--listen-onion`, `--i2p-accept-incoming`, Electrum, and Esplora: `ADD_ONION NEW` per service with the P2P virtual port on the loopback bind, each key saved `0600` under `onion/`, each SAM destination under `i2p/`, `STREAM FORWARD` to each port, and `getnetworkinfo.localaddresses` lists the three onions and the I2P address. A SAFECOOKIE restart reuses every saved key and destination | diff --git a/changelog.d/esplora-nullcount-http.md b/changelog.d/esplora-nullcount-http.md new file mode 100644 index 000000000..d9dcdf8e6 --- /dev/null +++ b/changelog.d/esplora-nullcount-http.md @@ -0,0 +1,12 @@ +Fixed + +- **Esplora paged address history is served above `--max-sh-creates`.** + `/txs`, `/txs/chain`, and `/txs/summary` return their page when a script + has more creates than the cap. Unpaged stats and `/utxo` still return 503. +- **`/txs/summary` rows include `tx_position`**, and `?asof=` is accepted on + that route. The page stays 25 rows. +- **Esplora tx JSON omits an empty `witness`.** `inner_redeemscript_asm` is + only the P2SH redeem script, and `inner_witnessscript_asm` is only a + P2WSH witness script or a Taproot script-path leaf. +- **`/block/:hash/txs/:start` at or past the last tx is 404** + `start index out of range`. diff --git a/crates/rbitcoin-esplora/src/esplora_http_journey.rs b/crates/rbitcoin-esplora/src/esplora_http_journey.rs index 795c894e0..5654e7add 100644 --- a/crates/rbitcoin-esplora/src/esplora_http_journey.rs +++ b/crates/rbitcoin-esplora/src/esplora_http_journey.rs @@ -98,6 +98,7 @@ async fn wallet_pages_after_txid(addr: SocketAddr, sh1: &str) { assert_eq!(st, 200, "{body}"); let sum: Vec = serde_json::from_str(&body).unwrap(); assert_eq!(sum[0]["txid"], t1); + assert_eq!(sum[0]["tx_position"], 1, "pay sits after the coinbase: {sum:?}"); assert!(!sum.iter().any(|v| v["txid"] == t3)); let unknown = "ff".repeat(32); @@ -315,6 +316,19 @@ async fn junk_paths_asof_and_post(addr: SocketAddr, genesis: &[u8; 32]) { assert!(resp.contains(needle), "{path}: {resp}"); } } + + let sh = block_hash_hex(&script_hash(&[0x51])); + let (st, body) = http_get( + addr, + &format!("/scripthash/{sh}/txs/summary?asof={h0hex}"), + ) + .await; + assert_eq!(st, 200, "summary accepts ?asof=: {body}"); + let rows: Vec = serde_json::from_str(&body).unwrap(); + assert!( + rows.iter().any(|r| r["tx_position"] == 0), + "genesis coinbase is position 0: {body}" + ); } /// A same-height replace restamps a scripthash read. `/mempool` never pins a view. diff --git a/crates/rbitcoin-esplora/src/esplora_sh_journey.rs b/crates/rbitcoin-esplora/src/esplora_sh_journey.rs index 04a9880b6..12940a1ac 100644 --- a/crates/rbitcoin-esplora/src/esplora_sh_journey.rs +++ b/crates/rbitcoin-esplora/src/esplora_sh_journey.rs @@ -9,6 +9,25 @@ async fn cap_refuses_three_creates(q: &Query, app: &Router) { body.contains("scripthash join exceeds --max-sh-creates"), "{body}" ); + let (st, utxo) = + oneshot_http(app, get_with_client(&format!("/scripthash/{h51}/utxo"), "cap")).await; + assert_eq!(st, 503, "unpaged utxo stays capped: {utxo}"); + let (st, txs) = oneshot_http(app, get_with_client(&format!("/scripthash/{h51}/txs"), "cap")).await; + assert_eq!(st, 200, "paged /txs is served above the cap: {txs}"); + let (st, page) = oneshot_http( + app, + get_with_client(&format!("/scripthash/{h51}/txs/chain"), "cap"), + ) + .await; + assert_eq!(st, 200, "paged chain is served above the cap: {page}"); + let rows: Value = serde_json::from_str(&page).unwrap(); + assert!(!rows.as_array().unwrap().is_empty(), "{page}"); + let (st, sum) = oneshot_http( + app, + get_with_client(&format!("/scripthash/{h51}/txs/summary"), "cap"), + ) + .await; + assert_eq!(st, 200, "paged summary is served above the cap: {sum}"); q.set_max_sh_creates(0); } diff --git a/crates/rbitcoin-esplora/src/handlers.rs b/crates/rbitcoin-esplora/src/handlers.rs index b64036a52..c49ef0827 100644 --- a/crates/rbitcoin-esplora/src/handlers.rs +++ b/crates/rbitcoin-esplora/src/handlers.rs @@ -296,6 +296,9 @@ fn block_txs_impl(st: AppState, hash_hex: &str, start: u32) -> Response { }; match st.query.block_tx_fks(h) { Ok(fks) => { + if (start as usize) >= fks.len() { + return (StatusCode::NOT_FOUND, "start index out of range").into_response(); + } let page: Vec<_> = fks.into_iter().skip(start as usize).take(25).collect(); let mut out = Vec::with_capacity(page.len()); for fk in page { @@ -1089,6 +1092,7 @@ fn summaries_json( query: &Query, items: &[ScriptHashTxSummary], ) -> Result { + let mut spans: HashMap = HashMap::new(); let mut out = Vec::with_capacity(items.len()); for it in items { let time = if it.height >= 0 { @@ -1099,12 +1103,27 @@ fn summaries_json( } else { 0 }; - out.push(json!({ + let mut row = json!({ "txid": block_hash_hex(&it.txid), "value": it.value, "height": it.height, "time": time, - })); + }); + if it.height >= 0 && !it.tx_fk.is_null() { + let h = it.height as u32; + let span = match spans.entry(h) { + std::collections::hash_map::Entry::Occupied(e) => *e.get(), + std::collections::hash_map::Entry::Vacant(e) => { + *e.insert(query.tx_fk_span_at_height(Height(h))?) + } + }; + if let Some(pos) = + rbitcoin_store::HeaderTxsTable::index_in_span(span.0, span.1, it.tx_fk) + { + row["tx_position"] = json!(pos); + } + } + out.push(row); } Ok(Value::Array(out)) } diff --git a/crates/rbitcoin-esplora/src/server.rs b/crates/rbitcoin-esplora/src/server.rs index 41102cf74..242bc9634 100644 --- a/crates/rbitcoin-esplora/src/server.rs +++ b/crates/rbitcoin-esplora/src/server.rs @@ -170,8 +170,8 @@ fn path_uses_sh_view(path: &str) -> bool { || path.starts_with("/scripthashes/") } -/// COMPAT.md: `?asof=` only on tx status/outspend(s) and address/scripthash -/// `/`, `/utxo`, `/txs`, `/txs/chain` (not `/txs/mempool`). +/// COMPAT.md: `?asof=` on tx status/outspend(s) and address/scripthash +/// `/`, `/utxo`, `/txs`, `/txs/chain`, `/txs/summary` (not `/txs/mempool`). fn path_accepts_asof(path: &str) -> bool { let segs: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect(); matches!( @@ -189,6 +189,10 @@ fn path_accepts_asof(path: &str) -> bool { | ["scripthash", _, "txs", "chain"] | ["address", _, "txs", "chain", _] | ["scripthash", _, "txs", "chain", _] + | ["address", _, "txs", "summary"] + | ["scripthash", _, "txs", "summary"] + | ["address", _, "txs", "summary", _] + | ["scripthash", _, "txs", "summary", _] ) } @@ -1559,6 +1563,14 @@ mod tests { .route("/scripthash/{hash}", get(handlers::scripthash_info)) .route("/scripthash/{hash}/utxo", get(handlers::scripthash_utxo)) .route("/scripthash/{hash}/txs", get(handlers::scripthash_txs)) + .route( + "/scripthash/{hash}/txs/chain", + get(handlers::scripthash_txs_chain), + ) + .route( + "/scripthash/{hash}/txs/summary", + get(handlers::scripthash_txs_summary), + ) .route("/address/{addr}/utxo", get(handlers::address_utxo)) .route("/scripthashes/txs", post(handlers::post_scripthashes_txs)) .with_state(state) @@ -1680,6 +1692,9 @@ mod tests { assert!(path_accepts_asof("/tx/ab/outspend/0")); assert!(path_accepts_asof("/scripthash/ab/utxo")); assert!(path_accepts_asof("/address/bcrt1q/txs/chain/cd")); + assert!(path_accepts_asof("/address/bcrt1q/txs/summary")); + assert!(path_accepts_asof("/scripthash/ab/txs/summary/cd")); + assert!(!path_accepts_asof("/address/bcrt1q/txs/mempool")); assert!(!path_accepts_asof("/tx/ab")); assert!(!path_accepts_asof("/mempool")); assert!(!path_accepts_asof("/scripthash/ab/txs/mempool")); @@ -1906,6 +1921,29 @@ mod tests { rbitcoin_primitives::display_hash_hex(&txid.to_byte_array()) } + /// electrs: a start index at or past the last tx is 404, not an empty page. + #[tokio::test] + async fn block_txs_past_end_is_not_found() { + let (dir, q) = temp_query("txs-range"); + let (header, ta) = coinbase(0, Fk::NULL, None); + let hash = header.hash; + q.connect_block(Height(0), &header, &[ta]).unwrap(); + let cfg = EsploraConfig::with_network("127.0.0.1:0".parse().unwrap(), Network::Regtest); + let handle = run_esplora(cfg, Arc::new(q), None).await.expect("listen"); + let h = block_hash_hex(&hash); + let (st, body) = http_get(handle.local_addr, &format!("/block/{h}/txs/0")).await; + assert_eq!(st, 200, "{body}"); + let rows: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert_eq!(rows.as_array().map(|a| a.len()), Some(1), "{body}"); + let (st, body) = http_get(handle.local_addr, &format!("/block/{h}/txs/25")).await; + assert_eq!(st, 404, "{body}"); + assert!(body.contains("start index out of range"), "{body}"); + let (st, body) = http_get(handle.local_addr, &format!("/block/{h}/txs/1")).await; + assert_eq!(st, 400, "{body}"); + handle.shutdown().await; + let _ = std::fs::remove_dir_all(&dir); + } + include!("esplora_sh_journey.rs"); include!("esplora_http_journey.rs"); } diff --git a/crates/rbitcoin-esplora/src/tx_json.rs b/crates/rbitcoin-esplora/src/tx_json.rs index 390949e59..8e752f3cc 100644 --- a/crates/rbitcoin-esplora/src/tx_json.rs +++ b/crates/rbitcoin-esplora/src/tx_json.rs @@ -308,16 +308,13 @@ fn tx_json_from_wire( vin_obj["scriptsig"] = Value::String(ss_f.hex); vin_obj["scriptsig_asm"] = Value::String(ss_f.asm); - let wit: Vec = tin.witness.iter().map(hex_encode).collect(); - vin_obj["witness"] = json!(wit); - - if let Some(asm) = inner_redeemscript_asm(ss) { - vin_obj["inner_redeemscript_asm"] = Value::String(asm); - } - if let Some(asm) = inner_witnessscript_asm(&wit) { - vin_obj["inner_witnessscript_asm"] = Value::String(asm); + let wit_raw: Vec> = tin.witness.iter().map(|w| w.to_vec()).collect(); + if !wit_raw.is_empty() { + let wit: Vec = wit_raw.iter().map(hex_encode).collect(); + vin_obj["witness"] = json!(wit); } + let mut spk_for_inner: Option> = None; if is_coinbase { prev_spks.push(Vec::new()); } else if let Some(prev) = prevout_json(query, stored_inputs, i, tin, network, mempool)? { @@ -333,12 +330,16 @@ fn tx_json_from_wire( .and_then(|x| x.as_str()) .and_then(|h| rbitcoin_primitives::hex_decode(h).ok()) .unwrap_or_default(); + spk_for_inner = Some(spk.clone()); prev_spks.push(spk); vin_obj["prevout"] = prev; } else { fee_in = None; prev_spks.push(Vec::new()); } + if let Some(spk) = spk_for_inner.as_deref() { + attach_inner_scripts(&mut vin_obj, spk, ss, &wit_raw); + } vin.push(vin_obj); } @@ -440,33 +441,60 @@ fn block_hash_hex(hash: &[u8; 32]) -> String { display_hash_hex(hash) } -/// Last push of scriptSig as redeemscript asm (P2SH). -fn inner_redeemscript_asm(script_sig: &[u8]) -> Option { - let last = last_push_data(script_sig)?; - if last.is_empty() { - return None; +/// electrs `get_innerscripts`: redeem only for P2SH; witness script only for +/// P2WSH (or a P2SH redeem that is P2WSH); Taproot leaf is the second-to-last +/// witness item after a trailing annex (`0x50`). Unknown prevout emits neither. +fn attach_inner_scripts(vin_obj: &mut Value, spk: &[u8], script_sig: &[u8], witness: &[Vec]) { + let prev = bitcoin::Script::from_bytes(spk); + let redeem = if prev.is_p2sh() { + last_push_data(script_sig).filter(|b| !b.is_empty() && b.len() <= 10_000) + } else { + None + }; + if let Some(bytes) = redeem { + if let Some(asm) = script_asm(bytes) { + vin_obj["inner_redeemscript_asm"] = Value::String(asm); + } + } + let nested = redeem + .map(bitcoin::Script::from_bytes) + .filter(|s| s.is_witness_program()); + let program = nested.unwrap_or(prev); + let leaf = if program.is_p2wsh() { + witness.last().map(Vec::as_slice) + } else if program.is_p2tr() { + tapscript_leaf(witness) + } else { + None + }; + if let Some(bytes) = leaf { + if let Some(asm) = script_asm(bytes) { + vin_obj["inner_witnessscript_asm"] = Value::String(asm); + } } - Some(bitcoin::Script::from_bytes(last).to_asm_string()) } -fn is_der_sig_prefix(bytes: &[u8]) -> bool { - bytes.first() == Some(&0x30) +fn script_asm(bytes: &[u8]) -> Option { + if bytes.is_empty() || bytes.len() > 10_000 { + return None; + } + Some(bitcoin::Script::from_bytes(bytes).to_asm_string()) } -/// Witness script: last stack item when it looks like a script (P2WSH / nested). -fn inner_witnessscript_asm(witness_hex: &[String]) -> Option { - if witness_hex.len() < 2 { +/// BIP341: a last witness item starting with `0x50` is the annex. The leaf +/// script is then the item before the control block. +fn tapscript_leaf(witness: &[Vec]) -> Option<&[u8]> { + let mut n = witness.len(); + if n == 0 { return None; } - let last = witness_hex.last()?; - let bytes = rbitcoin_primitives::hex_decode(last).ok()?; - if bytes.is_empty() || bytes.len() > 10_000 { - return None; + if witness[n - 1].first() == Some(&0x50) { + n -= 1; } - if is_der_sig_prefix(&bytes) { + if n < 2 { return None; } - Some(bitcoin::Script::from_bytes(&bytes).to_asm_string()) + Some(witness[n - 2].as_slice()) } fn last_push_data(script: &[u8]) -> Option<&[u8]> { @@ -545,23 +573,190 @@ mod tests { } #[test] - fn redeem_and_witness_script_asm_helpers() { - assert!(inner_redeemscript_asm(&[]).is_none()); - assert!(inner_redeemscript_asm(&[0x00]).is_none()); // empty push - let redeem = inner_redeemscript_asm(&[0x01, 0x51]).unwrap(); - assert!(redeem.contains("OP_1") || redeem.contains("1")); - - assert!(inner_witnessscript_asm(&[]).is_none()); - assert!(inner_witnessscript_asm(&[String::from("51")]).is_none()); // len < 2 - // Two items, last is OP_TRUE script — not a DER sig. - let asm = inner_witnessscript_asm(&[String::from("00"), String::from("51")]).unwrap(); - assert!(!asm.is_empty()); - // DER-looking last item skipped. - assert!(inner_witnessscript_asm(&[String::from("00"), String::from("3000")]).is_none()); - // Empty last stack item. - assert!(inner_witnessscript_asm(&[String::from("00"), String::new()]).is_none()); - // Invalid hex. - assert!(inner_witnessscript_asm(&[String::from("00"), String::from("zz")]).is_none()); + fn vin_inner_scripts_follow_prevout_type() { + use bitcoin::hashes::Hash; + use bitcoin::{ + absolute::LockTime, transaction::Version, Amount, OutPoint, ScriptBuf, Sequence, + Transaction, TxIn, TxOut, Witness, + }; + use rbitcoin_primitives::{Fk, Height}; + use rbitcoin_query::testutil::FixtureChain; + use rbitcoin_query::TxApply; + use rbitcoin_store::{HeaderRecord, InputRecord, OutputRecord, TxRecord}; + + let p2pkh = ScriptBuf::from_bytes(vec![ + 0x76, 0xa9, 0x14, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, + 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x88, 0xac, + ]); + let p2sh = ScriptBuf::from_bytes(vec![ + 0xa9, 0x14, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, + 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x22, 0x87, + ]); + let p2wpkh = ScriptBuf::from_bytes({ + let mut v = vec![0x00, 0x14]; + v.extend_from_slice(&[0x33; 20]); + v + }); + let witness_script = ScriptBuf::from_bytes(vec![0x51]); + let p2wsh = ScriptBuf::new_p2wsh(&witness_script.wscript_hash()); + let p2tr = + ScriptBuf::new_p2tr_tweaked(bitcoin::key::TweakedPublicKey::dangerous_assume_tweaked( + bitcoin::XOnlyPublicKey::from_slice(&[0x44; 32]).unwrap(), + )); + let spks = [p2pkh, p2sh, p2wpkh, p2wsh, p2tr]; + + let (dir, q) = rbitcoin_query::testutil::tiny_query_labeled("inner-scripts"); + let mut txid = [0u8; 32]; + txid[31] = 0xcb; + let parent = TxApply { + tx: TxRecord { + txid, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: spks.len() as u32, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![0x01], + witness: vec![], + }], + outputs: spks + .iter() + .map(|s| OutputRecord::unspent(10_000, s.to_bytes())) + .collect(), + }; + let header = HeaderRecord { + prev_fk: Fk::NULL, + version: 1, + timestamp: 1, + bits: 0x207fffff, + nonce: 0, + merkle_root: [1u8; 32], + hash: [1u8; 32], + size: 0, + weight: 0, + }; + q.connect_block(Height(0), &header, &[parent]).unwrap(); + + let prev = bitcoin::Txid::from_byte_array(txid); + let mut child = Transaction { + version: Version::TWO, + lock_time: LockTime::ZERO, + input: vec![ + TxIn { + previous_output: OutPoint { + txid: prev, + vout: 0, + }, + script_sig: ScriptBuf::from_bytes(vec![0x01, 0x51]), + sequence: Sequence::MAX, + witness: Witness::new(), + }, + TxIn { + previous_output: OutPoint { + txid: prev, + vout: 1, + }, + script_sig: ScriptBuf::from_bytes(vec![0x01, 0x51]), + sequence: Sequence::MAX, + witness: Witness::new(), + }, + TxIn { + previous_output: OutPoint { + txid: prev, + vout: 2, + }, + script_sig: ScriptBuf::new(), + sequence: Sequence::MAX, + witness: Witness::from_slice(&[&[0x30, 0x01][..], &[0x02, 0x55][..]]), + }, + TxIn { + previous_output: OutPoint { + txid: prev, + vout: 3, + }, + script_sig: ScriptBuf::new(), + sequence: Sequence::MAX, + witness: Witness::from_slice(&[&[0x00][..], witness_script.as_bytes()]), + }, + TxIn { + previous_output: OutPoint { + txid: prev, + vout: 4, + }, + script_sig: ScriptBuf::new(), + sequence: Sequence::MAX, + witness: Witness::from_slice(&[ + &[0x01][..], + &[0x51][..], + &{ + let mut c = vec![0xc0]; + c.extend_from_slice(&[0x44; 32]); + c + }[..], + ]), + }, + ], + output: vec![TxOut { + value: Amount::from_sat(1_000), + script_pubkey: ScriptBuf::new(), + }], + }; + let v = build_tx_json_from_tx(&q, &child, Network::Regtest, None, None).unwrap(); + let vin = v["vin"].as_array().unwrap(); + assert!(vin[0].get("witness").is_none(), "p2pkh {}", vin[0]); + assert!(vin[0].get("inner_redeemscript_asm").is_none(), "{}", vin[0]); + assert!( + vin[0].get("inner_witnessscript_asm").is_none(), + "{}", + vin[0] + ); + assert!( + vin[1]["inner_redeemscript_asm"] + .as_str() + .unwrap() + .contains("OP_1") + || vin[1]["inner_redeemscript_asm"] + .as_str() + .unwrap() + .contains('1'), + "p2sh {}", + vin[1] + ); + assert!( + vin[1].get("inner_witnessscript_asm").is_none(), + "{}", + vin[1] + ); + assert!(vin[2].get("witness").is_some(), "{}", vin[2]); + assert!( + vin[2].get("inner_witnessscript_asm").is_none(), + "p2wpkh {}", + vin[2] + ); + let wsh = vin[3]["inner_witnessscript_asm"].as_str().unwrap(); + assert!(wsh.contains("OP_1") || wsh.contains('1'), "p2wsh {wsh}"); + let tr = vin[4]["inner_witnessscript_asm"].as_str().unwrap(); + assert!(tr.contains("OP_1") || tr.contains('1'), "tapscript {tr}"); + assert!( + !tr.contains("OP_UNKNOWN"), + "control block must not be the leaf: {tr}" + ); + + child.input[4].witness = Witness::from_slice(&[&[0x22; 64][..]]); + let keypath = build_tx_json_from_tx(&q, &child, Network::Regtest, None, None).unwrap(); + assert!( + keypath["vin"][4].get("inner_witnessscript_asm").is_none(), + "key path {}", + keypath["vin"][4] + ); + let _ = std::fs::remove_dir_all(&dir); } #[test] diff --git a/crates/rbitcoin-query/src/query_sh_caps_journey.rs b/crates/rbitcoin-query/src/query_sh_caps_journey.rs index a8235a112..b336522fb 100644 --- a/crates/rbitcoin-query/src/query_sh_caps_journey.rs +++ b/crates/rbitcoin-query/src/query_sh_caps_journey.rs @@ -109,6 +109,35 @@ fn sh_history_caps() { "the newest row spends the oldest create" ); + let view = q.pin_sh_chain_view().unwrap().expect("sh view"); + let page = crate::scripthash::HistoryFilter::esplora_chain_page(None); + let mut slot = None; + let rows = q + .scripthash_history_filtered_slot_in(&sh, &page, &mut slot, &view) + .expect("paged history skips the unpaged cap"); + assert!(!rows.is_empty()); + assert!( + slot.is_none(), + "a paged miss must not fill the join the cap refuses" + ); + let sums = q + .scripthash_history_summary_filtered_slot_in(&sh, &page, &mut slot, &view) + .expect("paged summary skips the unpaged cap"); + assert!(!sums.is_empty()); + assert!(slot.is_none()); + let err = q + .scripthash_history_filtered_slot_in( + &sh, + &crate::scripthash::HistoryFilter::open(), + &mut slot, + &view, + ) + .unwrap_err(); + assert!( + matches!(err, StoreError::Rejected(m) if m == Query::MAX_SH_CREATES_MSG), + "{err}" + ); + q.set_max_sh_creates(0); assert_eq!(q.scripthash_history(&sh).unwrap().len(), 7); q.set_max_sh_creates(7); diff --git a/crates/rbitcoin-query/src/reconstruct.rs b/crates/rbitcoin-query/src/reconstruct.rs index fe63605b5..29e27ee89 100644 --- a/crates/rbitcoin-query/src/reconstruct.rs +++ b/crates/rbitcoin-query/src/reconstruct.rs @@ -199,6 +199,23 @@ impl Query { .ok_or(StoreError::Corrupt("confirmed header missing body list")) } + /// Contiguous Class A fk span for the confirmed block at `height`. + /// + /// Position in the block is [`rbitcoin_store::HeaderTxsTable::index_in_span`]. + /// This does not expand the block's fk list. A confirmed header with no + /// body span is the same corrupt miss as [`Self::block_tx_fks`]. + pub fn tx_fk_span_at_height(&self, height: Height) -> Result<(Fk, u32), QueryError> { + let header_fk = self + .store + .confirmed + .get(height)? + .ok_or(StoreError::NotFound)?; + self.store + .header_txs + .get_range(header_fk)? + .ok_or(StoreError::Corrupt("confirmed header missing body list")) + } + fn contiguous_fk_run(fks: &[Fk]) -> Option<(u64, u64)> { let first = fks.first()?.get()?; if first == 0 { diff --git a/crates/rbitcoin-query/src/scripthash.rs b/crates/rbitcoin-query/src/scripthash.rs index b2ebf472c..2cc4a710a 100644 --- a/crates/rbitcoin-query/src/scripthash.rs +++ b/crates/rbitcoin-query/src/scripthash.rs @@ -39,12 +39,15 @@ pub struct ScriptHashHistoryItem { pub fee: Option, } -/// Light history row (mempool.space `/txs/summary` shape without `time`). +/// Light history row (mempool.space `/txs/summary` shape without `time` +/// or `tx_position`; those are filled when the row is serialized). #[derive(Clone, Debug, PartialEq, Eq)] pub struct ScriptHashTxSummary { pub txid: [u8; 32], pub value: i64, pub height: i64, + /// Confirmed Class A fk. `NULL` when the row has no stored body. + pub tx_fk: rbitcoin_primitives::Fk, } /// Sort order for [`apply_history_filter`]. @@ -239,6 +242,7 @@ fn summaries_from_joined( txid: it.txid, value: net.get(&it.tx_fk).copied().unwrap_or(0), height: it.height, + tx_fk: it.tx_fk, }) .collect() } @@ -637,6 +641,32 @@ impl Query { Ok(out) } + /// A height window never wants the whole join. A page limit skips it only + /// when the unpaged join would hit `--max-sh-creates`, so an under-cap + /// page still fills the last-1 slot. + fn page_without_full_slot( + &self, + scripthash: &[u8; 32], + filter: &HistoryFilter, + slot: &Option>, + view: &ChainView, + ) -> Result { + if slot + .as_ref() + .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, view)) + { + return Ok(false); + } + if filter.to_height.is_some() { + return Ok(true); + } + if filter.limit.is_none() { + return Ok(false); + } + let cap = self.max_sh_creates(); + Ok(cap > 0 && self.scripthash_create_count(scripthash)? > cap) + } + fn sh_join_slot_hit(slot: &ShJoinSlot, scripthash: &[u8; 32], view: &ChainView) -> bool { slot.scripthash == *scripthash && slot.tip_hash == view.hash } @@ -1010,10 +1040,7 @@ impl Query { slot: &mut Option>, view: &ChainView, ) -> Result, QueryError> { - let hit = slot - .as_ref() - .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, view)); - if !hit && filter.to_height.is_some() { + if self.page_without_full_slot(scripthash, filter, slot, view)? { return self.scripthash_history_filtered_in(scripthash, filter, view); } self.ensure_sh_join_slot_in(scripthash, slot, view)?; @@ -1049,10 +1076,7 @@ impl Query { slot: &mut Option>, view: &ChainView, ) -> Result, QueryError> { - let hit = slot - .as_ref() - .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, view)); - if !hit && filter.to_height.is_some() { + if self.page_without_full_slot(scripthash, filter, slot, view)? { return self.scripthash_history_summary_filtered_in(scripthash, filter, view); } self.ensure_sh_join_slot_in(scripthash, slot, view)?; diff --git a/crates/rbitcoin-store/src/chain.rs b/crates/rbitcoin-store/src/chain.rs index 2c0261ac2..096574dbc 100644 --- a/crates/rbitcoin-store/src/chain.rs +++ b/crates/rbitcoin-store/src/chain.rs @@ -978,6 +978,12 @@ mod chain_table_tests { assert!(!ht.contains_tx(Fk(1), Fk(9)).unwrap()); assert!(!ht.contains_tx(Fk(1), Fk(13)).unwrap()); assert!(!ht.contains_tx(Fk(2), Fk(10)).unwrap(), "other header"); + assert_eq!(ht.tx_index(Fk(1), Fk(10)).unwrap(), Some(0)); + assert_eq!(ht.tx_index(Fk(1), Fk(12)).unwrap(), Some(2)); + assert_eq!(ht.tx_index(Fk(1), Fk(9)).unwrap(), None); + assert_eq!(ht.tx_index(Fk(1), Fk(13)).unwrap(), None); + assert_eq!(ht.tx_index(Fk(1), Fk::NULL).unwrap(), None); + assert_eq!(ht.tx_index(Fk(2), Fk(10)).unwrap(), None, "other header"); let _ = std::fs::remove_dir_all(&dir); } } @@ -1123,18 +1129,33 @@ impl HeaderTxsTable { Ok(self.get_range(header_fk)?.is_some()) } + /// Index of `tx_fk` inside a contiguous `(first, count)` span. + pub fn index_in_span(first: Fk, n: u32, tx_fk: Fk) -> Option { + let tid = tx_fk.get()?; + let lo = first.get()?; + if n == 0 { + return None; + } + let end = lo.saturating_add(u64::from(n)); + if tid < lo || tid >= end { + return None; + } + u32::try_from(tid - lo).ok() + } + /// True if `tx_fk` is in this header's contiguous Class A body range. pub fn contains_tx(&self, header_fk: Fk, tx_fk: Fk) -> Result { - let Some(tid) = tx_fk.get() else { - return Ok(false); - }; + Ok(self.tx_index(header_fk, tx_fk)?.is_some()) + } + + /// Index of `tx_fk` in this header's contiguous Class A range. + /// + /// One `(first, count)` read. Does not expand the block's fk list. + pub fn tx_index(&self, header_fk: Fk, tx_fk: Fk) -> Result, StoreError> { let Some((first, n)) = self.get_range(header_fk)? else { - return Ok(false); - }; - let Some(lo) = first.get() else { - return Ok(false); + return Ok(None); }; - Ok(tid >= lo && tid < lo.saturating_add(u64::from(n))) + Ok(Self::index_in_span(first, n, tx_fk)) } /// Drop Class A body association for `header_fk` (does not free tx rows). diff --git a/crates/rbitcoin-store/src/lib.rs b/crates/rbitcoin-store/src/lib.rs index dd789661a..a8248b3b1 100644 --- a/crates/rbitcoin-store/src/lib.rs +++ b/crates/rbitcoin-store/src/lib.rs @@ -64,6 +64,7 @@ mod txstat; mod uring_session; mod var_table; +pub use crate::chain::HeaderTxsTable; pub use crate::compact::output_flags; pub use address_head::{is_probe_exhausted_error, is_store_corrupt_display}; pub use block_filter::{BlockFilterRecord, BlockFilterSlot, BlockFilterTable, StoredBlockFilter}; diff --git a/crates/rbitcoin-test/tests/cross_surface.rs b/crates/rbitcoin-test/tests/cross_surface.rs index e816d9372..bfc7af855 100644 --- a/crates/rbitcoin-test/tests/cross_surface.rs +++ b/crates/rbitcoin-test/tests/cross_surface.rs @@ -553,10 +553,12 @@ async fn pin_esplora_block_txs_pages(esplora_addr: SocketAddr, tip_hash: &str, n assert_eq!(st, 404, "unknown block txs: {body}"); let next = u32::try_from((n_tx / 25 + 1) * 25).expect("txs start"); - let (st, body, empty) = - esplora_json_array(esplora_addr, &format!("/block/{tip_hash}/txs/{next}")).await; - assert_eq!(st, 200, "one-past last page: {body}"); - assert!(empty.is_empty(), "Esplora empty page is [] not 404: {body}"); + let (st, body) = http_get(esplora_addr, &format!("/block/{tip_hash}/txs/{next}")).await; + assert_eq!(st, 404, "one-past last page: {body}"); + assert!( + body.contains("start index out of range"), + "one-past last page: {body}" + ); } /// Extra Esplora HTTP leftover: txids, merkle-proof, coinbase outspend.