esplora: paged history, summary fields, and tx JSON

COMPAT.md

@@ -226,7 +226,7 @@ App `ServeLimits` always on (same model as Electrum).
 | Block | done | `/block/:hash` JSON, `/raw`, `/status`, `/header`, `/txids`, `/txid/:i`, `/txs[/:start]`. JSON `bits` is the compact-target **u32** (Esplora schema, not Core hex). `size` / `weight` are BIP144 total size and BIP141 weight (witness included). |
 | Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` (`vin` from the spent slot; unspent omits it). Below seqsigwit prune: JSON is 200 partial (`pruned: true`, vout+status, empty `vin`); stamped `txstat` fills `fee`/`size`/`weight`. Mempool-only txs (not in Class A) use the wire body from the mempool hub (`vin`/`vout`/`size`/`weight`/`fee`/`sigops`, `status.confirmed` false) including `GET /tx/:txid/status`. Live `/outspend(s)` overlay mempool spends of confirmed coins; `?asof=` omits mempool. `?asof=<hash>` on `/status` and `/outspend(s)`: confirmed/spent as of that ancestor; 404 if not on chain. |
 | Address / scripthash | done | stats + `/utxo` + `/txs` + `/txs/mempool` + `/txs/chain[/:last_seen_txid]` + `/txs/summary[/:last_seen_txid]` (dialect; next row). `/utxo` matches Electrum listunspent (mempool funding + drop mempool-spent confirmed); `/txs` and `/txs/mempool` use full Esplora tx JSON for mempool-only rows (wire from the hub). `?after_txid=` on `/txs` and `/txs/summary` skips through that tx (mempool then chain); unknown or unparseable → **422** `after_txid not found`. `POST /addresses/txs` and `POST /scripthashes/txs` (and `/txs/summary`) take a JSON array (max **300**; over → **422** `body too long`); unique scripts join once per request, merge newest-first. Unix/loopback `X-Rbitcoin-Client`: last-1 GET join + last-bulk POST (16 MiB packed/client), 30s idle, 256 clients, singleflight on the same `(client, sh)`. Public TCP ignores the header (stack-local join). Concurrent different GET scripts replace last-1. Unbounded process LRU stays **X-M3**. Needs SH finalize. Stamp is visible SH (durable + pending write-behind), matching live tip while jobs sit in RAM. `?asof=<hash>` on `/`, `/utxo`, `/txs`, `/txs/chain`, `/txs/summary`: confirmed join at that ancestor **at or behind visible SH**, **no** mempool; headers are the asof hash; 404 if not on chain or ahead of visible SH. |
-| `/txs/summary` | dialect | **Not** in Blockstream Esplora [`API.md`](https://github.com/Blockstream/esplora/blob/master/API.md). Compact `{txid, value, height, time}` like mempool.space `/address/:addr/txs/summary`. Confirmed only (25/page, newest first); path cursor `/:last_seen_txid` and mempool.space `?after_txid=` (unknown → **422**). `value` is net sats for that script in that tx (funded − spent). `time` is the confirming header timestamp (`0` if the header is missing). Mempool rows stay on `/txs` and `/txs/mempool`. Over `--max-sh-creates` → **503**. |
+| `/txs/summary` | dialect | **Not** in Blockstream Esplora [`API.md`](https://github.com/Blockstream/esplora/blob/master/API.md). Compact `{txid, value, height, time, tx_position}` like mempool.space `/address/:addr/txs/summary`. Confirmed only (25/page, newest first); path cursor `/:last_seen_txid` and mempool.space `?after_txid=` (unknown → **422**). `value` is net sats for that script in that tx (funded − spent). `time` is the confirming header timestamp (`0` if the header is missing). `tx_position` is that tx's index in the block. Mempool rows stay on `/txs` and `/txs/mempool`. A page is served above `--max-sh-creates`; unpaged stats and `/utxo` still **503**. |
 | Mempool / fees | done | `/mempool`, `/mempool/txids`, `/mempool/txids/page[/:last]`, `/mempool/recent` (accept-order ring), `/fee-estimates` (sat/vB at the estimator's 1 sat/kvB resolution; a target without an estimate is left out, and **503** `fee estimates unavailable` when none has one). mempool.space `/api/v1/fees/recommended` is its backend's surface, not served here (**404**) |
 | `POST /tx` | done | broadcast via mempool hub; **503** if hub absent. `GET /broadcast?tx=` is the same admit path (legacy electrs). |
 | `POST /txs/test` | done | JSON array of hex (max 25) → `MempoolHub::test_accept` (no admit). `?maxfeerate=` is BTC/kvB like electrs (`0` unlimited; omitted → 0.1 BTC/kvB). |

TESTING.md

@@ -342,7 +342,7 @@ Prefer **one high-level scenario** per behavior cluster. Delete lower-level test
 | `electrum_tweaks_subscribe_streams_then_done` | Electrum | Cake `tweaks.subscribe`: one-height result, per-height notifies, `done`, and BIP352 hash-bind on a P2WPKH→P2TR spend. Keep zero-chunk / pre-taproot units |
 | `electrum_max_connections_rejects_extra_client` | Electrum | TCP cap drops the extra client |
 | `electrum_idle_timeout_disconnects_quiet_client` | Electrum | Idle timeout closes a quiet socket |
-| `esplora_broadcast_visible_in_rpc_and_electrum` | Node + Electrum + Esplora + RPC | One `run_p2p` datadir: HTTP `sendrawtransaction` / `testmempoolaccept` (allowed, missing-or-spent, exact 100 sat/kvB min-relay accept + one-sat-under reject, RBF one-sat-short incremental reject + exact incremental accept); Esplora `POST /tx` parent and mempool child appear in `getrawmempool` and Electrum mempool/history (`fee` on unconfirmed, including child `height = -1`); Electrum `listunspent` of that child is `height=-1` and the parent UTXO drops; process `gettxout` / `getchaintips`; Esplora `POST /txs/package` 1p1c (including parent-alone below min-relay + paying child), 25-tx accept, 26-tx and over-weight `package too large`; serving-only `submitpackage` refuses (relay off); live `GET /mempool` / `/mempool/txids` / `/mempool/recent` / `/fee-estimates` answers 503 while flow is cold and the chain holds too little fee history (a thin live pool alone sets no rate); process `getmempoolancestors` / descendants / cluster / `gettxspendingprevout` / feerate diagram / verbose `getrawmempool` on that 1p1c; `waitforblockheight` timeout=0 while behind returns the live tip; GBT stale `longpollid` is immediate; current id / `waitfornewblock` / `waitforblockheight` wake on the pad `generate`; `getblockhash` tip ok / tip+1 `-8`; unknown `getblock` `-5`; verbosity 0 hex and 2 vin/vout; `GET /blocks` 10 newest, `/blocks/0` and `/blocks/:tip` (start past tip clamps); `/block/:hash/txs/:start` last page shorter than 25, one-past last page `[]` (not 404), unknown hash 404; `/block/:hash/txids` + coinbase merkle-proof + unspent `outspend/0`; `/tx/:id/outspends`; `/block` JSON/raw/status/`txid/0` (OOB 404); `/tx/:id/raw` vs hex; merkleblock-proof; `/block-height` (missing 404); `/block/:hash/header` 160 hex; `/tx/:id/status` + full JSON (`unknown` OP_TRUE type, coinbase vin) and missing-tx 404s; OP_TRUE scripthash info/summary/utxo/`txs/chain` cursor and combined `/txs`. Keep crate no-hub mempool/fees/POST 503, reconstruct meters, header wire match, and `tx_status_json`; Esplora `/tx/:id/status` confirms the package parent on generate; `generate` includes those txs (parent before child) then leaves IBD (relay on); `scantxoutset` drops the spent coinbase and still sees a non-coinbase unspent; `submitpackage` maxfeerate reject, 1p1c success, already-in-mempool continue, below-min-relay parent + paying child success, 26-tx / over-weight `package too large`; immature coinbase sendraw rejects. Keep `accept.rs` reject units, package JSON errors, RPC dry-run orphan-count, leftover `gettxout` include_mempool / disconnected / leftover, `generate_selects_chained_mempool_parent_first`, `submitpackage_child_fail_keeps_parent`, maxburn `submitpackage`, and wait-on-stop units. Unix `--rpc-socket` (mode 0660, no datadir `rpc.sock`) `getblockcount` without Authorization; TCP `GET`/`POST /internal/mempool/txs`; `GET /internal/block/:hash/txs` full list vs public 25/page; `POST /internal/txs/outspends/by-txid` same-length unknown `[]` slot; `GET /address-prefix/bc1` **404**; unauthenticated Core REST on the RPC listener (`chaininfo`, block hash/headers/block/tx, mempool info/contents, `getutxos`, `deploymentinfo`, basic `blockfilter` bin/hex/json) and `getblockfilter` with `--block-filter-index` |
+| `esplora_broadcast_visible_in_rpc_and_electrum` | Node + Electrum + Esplora + RPC | One `run_p2p` datadir: HTTP `sendrawtransaction` / `testmempoolaccept` (allowed, missing-or-spent, exact 100 sat/kvB min-relay accept + one-sat-under reject, RBF one-sat-short incremental reject + exact incremental accept); Esplora `POST /tx` parent and mempool child appear in `getrawmempool` and Electrum mempool/history (`fee` on unconfirmed, including child `height = -1`); Electrum `listunspent` of that child is `height=-1` and the parent UTXO drops; process `gettxout` / `getchaintips`; Esplora `POST /txs/package` 1p1c (including parent-alone below min-relay + paying child), 25-tx accept, 26-tx and over-weight `package too large`; serving-only `submitpackage` refuses (relay off); live `GET /mempool` / `/mempool/txids` / `/mempool/recent` / `/fee-estimates` answers 503 while flow is cold and the chain holds too little fee history (a thin live pool alone sets no rate); process `getmempoolancestors` / descendants / cluster / `gettxspendingprevout` / feerate diagram / verbose `getrawmempool` on that 1p1c; `waitforblockheight` timeout=0 while behind returns the live tip; GBT stale `longpollid` is immediate; current id / `waitfornewblock` / `waitforblockheight` wake on the pad `generate`; `getblockhash` tip ok / tip+1 `-8`; unknown `getblock` `-5`; verbosity 0 hex and 2 vin/vout; `GET /blocks` 10 newest, `/blocks/0` and `/blocks/:tip` (start past tip clamps); `/block/:hash/txs/:start` last page shorter than 25, one-past last page 404, unknown hash 404; `/block/:hash/txids` + coinbase merkle-proof + unspent `outspend/0`; `/tx/:id/outspends`; `/block` JSON/raw/status/`txid/0` (OOB 404); `/tx/:id/raw` vs hex; merkleblock-proof; `/block-height` (missing 404); `/block/:hash/header` 160 hex; `/tx/:id/status` + full JSON (`unknown` OP_TRUE type, coinbase vin) and missing-tx 404s; OP_TRUE scripthash info/summary/utxo/`txs/chain` cursor and combined `/txs`. Keep crate no-hub mempool/fees/POST 503, reconstruct meters, header wire match, and `tx_status_json`; Esplora `/tx/:id/status` confirms the package parent on generate; `generate` includes those txs (parent before child) then leaves IBD (relay on); `scantxoutset` drops the spent coinbase and still sees a non-coinbase unspent; `submitpackage` maxfeerate reject, 1p1c success, already-in-mempool continue, below-min-relay parent + paying child success, 26-tx / over-weight `package too large`; immature coinbase sendraw rejects. Keep `accept.rs` reject units, package JSON errors, RPC dry-run orphan-count, leftover `gettxout` include_mempool / disconnected / leftover, `generate_selects_chained_mempool_parent_first`, `submitpackage_child_fail_keeps_parent`, maxburn `submitpackage`, and wait-on-stop units. Unix `--rpc-socket` (mode 0660, no datadir `rpc.sock`) `getblockcount` without Authorization; TCP `GET`/`POST /internal/mempool/txs`; `GET /internal/block/:hash/txs` full list vs public 25/page; `POST /internal/txs/outspends/by-txid` same-length unknown `[]` slot; `GET /address-prefix/bc1` **404**; unauthenticated Core REST on the RPC listener (`chaininfo`, block hash/headers/block/tx, mempool info/contents, `getutxos`, `deploymentinfo`, basic `blockfilter` bin/hex/json) and `getblockfilter` with `--block-filter-index` |
 | `fee_history_backfills_from_the_chain_when_relay_starts` | Node + RPC | `run_p2p` on a mature regtest datadir, started twice; `generate` leaves IBD and turns relay on; the preload writes the fee history file and the restart extends it; with flow cold and too little history, `estimatesmartfee` answers Core's insufficient-data shape. Rates from a ready history are the hub's `far_horizon_follows_block_history_not_pool_tail`; the success object is `smart_fee_json` |
 | `node_listen_and_exit` | Node + Electrum + Esplora + RPC | One `run_p2p` datadir, restarted with its one `--connect` refusing (a pinned connect at genesis still enters tip mode): a taken `--health-listen` port stops the start before the store opens; a junk `peers` file and a missing `--asmap` start an empty book and exit, and the saved book records the refused connect; the next start loads that book and a valid `ip_asn.dat`, and Esplora, Electrum, and RPC answer at genesis until `stop`. On that start `/healthz` is 200 while genesis `/readyz` is initial block download and `/metrics` reports the same; `generate` makes `/readyz` 200 and the scrape ready at height 1. An Electrum port and an RPC port another process holds warn, `/readyz` names both, and `/metrics` is absent without `--metrics`. Without `--connect` and with seeds on, regtest resolves none and the node exits short of tip mode; after a `--prune-seqsigwit` start, an unpruned start refuses. Live peers are `node_run_p2p_short`. The `/readyz` phase, lag, and tip-age table stays `readiness_reports_the_first_failing_gate`. A timed-out probe keeping its permit, the full-gate 503, and the routes answering again are `health_gate_caps` |
 | `tor_control_onion_lifecycle` | Node + RPC | One `run_p2p` datadir against a fake Tor control port and SAM bridge (live Tor and i2pd are overlay-functional). A cookie from another Tor (SAFECOOKIE server hash mismatch), a 2-byte cookie, and a Tor that offers only plain COOKIE each refuse the start, and none sends `AUTHENTICATE`. Password auth with `--listen-onion`, `--i2p-accept-incoming`, Electrum, and Esplora: `ADD_ONION NEW` per service with the P2P virtual port on the loopback bind, each key saved `0600` under `onion/`, each SAM destination under `i2p/`, `STREAM FORWARD` to each port, and `getnetworkinfo.localaddresses` lists the three onions and the I2P address. A SAFECOOKIE restart reuses every saved key and destination |

changelog.d/esplora-nullcount-http.md

@@ -0,0 +1,12 @@
+Fixed
+
+- **Esplora paged address history is served above `--max-sh-creates`.**
+  `/txs`, `/txs/chain`, and `/txs/summary` return their page when a script
+  has more creates than the cap. Unpaged stats and `/utxo` still return 503.
+- **`/txs/summary` rows include `tx_position`**, and `?asof=` is accepted on
+  that route. The page stays 25 rows.
+- **Esplora tx JSON omits an empty `witness`.** `inner_redeemscript_asm` is
+  only the P2SH redeem script, and `inner_witnessscript_asm` is only a
+  P2WSH witness script or a Taproot script-path leaf.
+- **`/block/:hash/txs/:start` at or past the last tx is 404**
+  `start index out of range`.

crates/rbitcoin-esplora/src/esplora_http_journey.rs

@@ -98,6 +98,7 @@ async fn wallet_pages_after_txid(addr: SocketAddr, sh1: &str) {
     assert_eq!(st, 200, "{body}");
     let sum: Vec<Value> = serde_json::from_str(&body).unwrap();
     assert_eq!(sum[0]["txid"], t1);
+    assert_eq!(sum[0]["tx_position"], 1, "pay sits after the coinbase: {sum:?}");
     assert!(!sum.iter().any(|v| v["txid"] == t3));
 
     let unknown = "ff".repeat(32);
@@ -315,6 +316,19 @@ async fn junk_paths_asof_and_post(addr: SocketAddr, genesis: &[u8; 32]) {
             assert!(resp.contains(needle), "{path}: {resp}");
         }
     }
+
+    let sh = block_hash_hex(&script_hash(&[0x51]));
+    let (st, body) = http_get(
+        addr,
+        &format!("/scripthash/{sh}/txs/summary?asof={h0hex}"),
+    )
+    .await;
+    assert_eq!(st, 200, "summary accepts ?asof=: {body}");
+    let rows: Vec<Value> = serde_json::from_str(&body).unwrap();
+    assert!(
+        rows.iter().any(|r| r["tx_position"] == 0),
+        "genesis coinbase is position 0: {body}"
+    );
 }
 
 /// A same-height replace restamps a scripthash read. `/mempool` never pins a view.

crates/rbitcoin-esplora/src/esplora_sh_journey.rs

@@ -9,6 +9,25 @@ async fn cap_refuses_three_creates(q: &Query, app: &Router) {
         body.contains("scripthash join exceeds --max-sh-creates"),
         "{body}"
     );
+    let (st, utxo) =
+        oneshot_http(app, get_with_client(&format!("/scripthash/{h51}/utxo"), "cap")).await;
+    assert_eq!(st, 503, "unpaged utxo stays capped: {utxo}");
+    let (st, txs) = oneshot_http(app, get_with_client(&format!("/scripthash/{h51}/txs"), "cap")).await;
+    assert_eq!(st, 200, "paged /txs is served above the cap: {txs}");
+    let (st, page) = oneshot_http(
+        app,
+        get_with_client(&format!("/scripthash/{h51}/txs/chain"), "cap"),
+    )
+    .await;
+    assert_eq!(st, 200, "paged chain is served above the cap: {page}");
+    let rows: Value = serde_json::from_str(&page).unwrap();
+    assert!(!rows.as_array().unwrap().is_empty(), "{page}");
+    let (st, sum) = oneshot_http(
+        app,
+        get_with_client(&format!("/scripthash/{h51}/txs/summary"), "cap"),
+    )
+    .await;
+    assert_eq!(st, 200, "paged summary is served above the cap: {sum}");
     q.set_max_sh_creates(0);
 }
 

crates/rbitcoin-esplora/src/handlers.rs

@@ -296,6 +296,9 @@ fn block_txs_impl(st: AppState, hash_hex: &str, start: u32) -> Response {
     };
     match st.query.block_tx_fks(h) {
         Ok(fks) => {
+            if (start as usize) >= fks.len() {
+                return (StatusCode::NOT_FOUND, "start index out of range").into_response();
+            }
             let page: Vec<_> = fks.into_iter().skip(start as usize).take(25).collect();
             let mut out = Vec::with_capacity(page.len());
             for fk in page {
@@ -1089,6 +1092,7 @@ fn summaries_json(
     query: &Query,
     items: &[ScriptHashTxSummary],
 ) -> Result<Value, rbitcoin_query::QueryError> {
+    let mut spans: HashMap<u32, (rbitcoin_primitives::Fk, u32)> = HashMap::new();
     let mut out = Vec::with_capacity(items.len());
     for it in items {
         let time = if it.height >= 0 {
@@ -1099,12 +1103,27 @@ fn summaries_json(
         } else {
             0
         };
-        out.push(json!({
+        let mut row = json!({
             "txid": block_hash_hex(&it.txid),
             "value": it.value,
             "height": it.height,
             "time": time,
-        }));
+        });
+        if it.height >= 0 && !it.tx_fk.is_null() {
+            let h = it.height as u32;
+            let span = match spans.entry(h) {
+                std::collections::hash_map::Entry::Occupied(e) => *e.get(),
+                std::collections::hash_map::Entry::Vacant(e) => {
+                    *e.insert(query.tx_fk_span_at_height(Height(h))?)
+                }
+            };
+            if let Some(pos) =
+                rbitcoin_store::HeaderTxsTable::index_in_span(span.0, span.1, it.tx_fk)
+            {
+                row["tx_position"] = json!(pos);
+            }
+        }
+        out.push(row);
     }
     Ok(Value::Array(out))
 }