diff --git a/COMPAT.md b/COMPAT.md index 488aee424..a72415040 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -20,7 +20,7 @@ backends**: in-process **Electrum** (confirmed + unconfirmed, libre-relay-class admission) and optional **Esplora-compatible REST** for the same role (history, UTXO, broadcast, block/tx fetch by id). Optional **Core-class JSON-RPC subset** (see [`docs/rpc.md`](./docs/rpc.md)) — not full Core wallet / mining parity. -**Scripthash index (`--shindex`) defaults off**. Electrum/Esplora **start without it**; address/scripthash methods fail closed (`scripthash index disabled`). Txid/outpoint/block/broadcast/fees work. Which wallets use that surface: [`docs/wallets.md`](./docs/wallets.md). Lightning chain sources: [`docs/lightning.md`](./docs/lightning.md). +**Scripthash index (`--sh-index`) defaults off**. Electrum/Esplora **start without it**; address/scripthash methods fail closed (`scripthash index disabled`). Txid/outpoint/block/broadcast/fees work. Which wallets use that surface: [`docs/wallets.md`](./docs/wallets.md). Lightning chain sources: [`docs/lightning.md`](./docs/lightning.md). On/off costs and start/IBD/tip behavior: [`OPERATOR.md`](./OPERATOR.md) (Scripthash index). Disable later leaves SH files on disk; follow does not wait on SH materialize. @@ -73,12 +73,12 @@ Full `/tx/:txid` JSON still has `vin[]`. Electrum has no outspend-vin surface. | Pruning / GUI | Unpruned: `seqsigwit.body`. `--prune-seqsigwit`: watermark + `NETWORK_LIMITED`, kept witness is 288 height files under `store/seqsigwit.window/` plus a RAM cap (`--prune-seqsigwit-ram-threshold-bytes`, `0` = files only). Not a rolling stem. Not Core `-prune` of headers/txout | Supported | | Mining template RPC | `getblocktemplate` / `getmininginfo` / `prioritisetransaction` (selector; no stratum) | GBT + stratum / pool stack | | SV2 Template Distribution | In-process TDP v2 server over Noise NX (`--sv2-tp-listen`, default off): `NewTemplate` / `SetNewPrevHash` per tip, `NewTemplate` on a same-tip fee gain (`--sv2-tp-fee-delta`, `--sv2-tp-template-interval`, sv2-tp defaults), `RequestTransactionData`, `SubmitSolution` → block accept. No templates during IBD; every template on the tip kept, up to 64 per session. No plaintext, no pool / JD roles | No TDP in Core; sv2-apps `bitcoin-core-sv2` bridges Core's IPC mining interface to TDP | -| Wallets | Electrum clients (requires `--shindex`) | Descriptor + legacy | -| Scripthash index | Optional (`--shindex`, default **off**); bulk at tip when on | External ElectrumX / Fulcrum; Core `-txindex` is different (txid→block) | +| Wallets | Electrum clients (requires `--sh-index`) | Descriptor + legacy | +| Scripthash index | Optional (`--sh-index`, default **off**); bulk at tip when on | External ElectrumX / Fulcrum; Core `-txindex` is different (txid→block) | | JSON-RPC | Documented **subset** ([`docs/rpc.md`](./docs/rpc.md)); Bearer token or opt-in Core cookie (`--rpc-cookie-file`), no `--rpcuser` / `--rpcpassword`; `rbitcoin-cli`. `--rpc-work-queue` defaults to **16** (HTTP occupancy, 503 when full; **0** is that default); a JSON-RPC array is one POST | Full Core RPC; `-rpcworkqueue` is in-flight HTTP jobs (503) | | GetData serve | Every hash is served in order. While **16** (`MAX_SERVE_BLOCKS`) getdata bodies or 4 MiB sit in the peer's queue (RAM cap), the rest waits on the session and no new message is read; none is dropped. Pings and the ping timeout still run | Core `ProcessGetData` pauses on `fPauseSend` and serves the rest before the next message | | Inbound eviction victim | After Core-shaped protect (netgroup / recent block / recent tx / min-ping / 8 longest-connected), disconnect the **newest** inbound in the largest remaining netgroup. A fully protected set rejects the new inbound | Core `SelectNodeToEvict` youngest in the oldest netgroup | -| `--sptweaks-dust` | Serve-time floor default **1000** sat (omit P2TR outs `value <=` floor). **546** matches Cake electrs. Not Cake/Electrum protocol | n/a (Electrum tweaks are not Core) | +| `--sp-tweaks-dust` | Serve-time floor default **1000** sat (omit P2TR outs `value <=` floor). **546** matches Cake electrs. Not Cake/Electrum protocol | n/a (Electrum tweaks are not Core) | | IBD empty-headers EOF | Empty `headers` to **our locator** + idle path latches `headers_done` even if a peer advertises a taller less-work height / junk `version.start_height`. A proven header walk at the tip that every taller peer failed to extend also ends IBD | Core header sync follows most-work; advertised `start_height` is not a remaining header count | Compact reconstruct fills short-ids from the live mempool graph, the @@ -121,7 +121,7 @@ Per-method notes, auth, and the shindex matrix live in | Index / tips (`getindexinfo`, `getchaintips`, `waitforblock*`) | done (`txindex` = Class A reconstruct) | | Fee (`estimatesmartfee`) | done (**10-minute inclusion** — not Core historical) | | Decode (`decoderawtransaction`, `decodescript`, `validateaddress`) | done (node subset; official Core dialect scripts stay `rpc-dialect`) | -| Regtest `generatetoaddress` / `generatetodescriptor` / `generateblock` / `generate` / `submitblock` / `setmocktime` | harness (regtest only except `submitblock`) | +| Regtest `generatetoaddress` / `generatetodescriptor` / `generateblock` / `generate` / `submitblock` / `setmocktime` | harness (`generate*` also on signet whose challenge is exactly `OP_TRUE`; `setmocktime` / `mockscheduler` stay regtest; `submitblock` is all networks) | | `invalidateblock` / `reconsiderblock` / `preciousblock` | done | | Mining template (`getblocktemplate`, `getmininginfo`, `prioritisetransaction`, `getmempoolcluster`) | done (no stratum / BIP9 testdummy / wallet keys) | | Wallet RPC; `createrawtransaction` / `combinerawtransaction`; `gettxoutsetinfo` | **never** (no UTXO set). The functional harness shims `gettxoutsetinfo` for `rpc_scantxoutset.py` | @@ -131,13 +131,13 @@ Per-method notes, auth, and the shindex matrix live in | Method | Status | Notes | |--------|--------|-------| | server.version / banner / features | done | Banner: libre-relay-class. `server.version[0]` is `rbitcoin-electrs ` — **not electrs**; see below. `server.version` negotiates: omitted → `1.6`; `"1.4"` → `1.4`; `["1.4","1.4.2"]` → `1.4.2`; `["1.4","1.6"]` → `1.6`; `"1.4.2-asof"` (or a range containing it) → as-of dialect. First call wins. `features.protocol_max` is `1.6`; `features.asof_protocol` is `1.4.2-asof`. `features.genesis_hash` is display-order hex (refuse a wrong-chain server before a tweaks scan). `features.tweaks` / `silent_payments` advertise the method; they are not a substitute for the stream. | -| blockchain.tweaks.subscribe | done | **Stream**, not a one-shot result: JSON-RPC `result` is the **first** height only; remaining heights are unsolicited notifications; `{"message":"done"}` ends the **chunk** (60s wall at a wave boundary, or the requested `count` if sooner) — Cake electrs clamps `count` to 1000 instead. Cake Wallet resubscribes from `syncHeight+1` ([cake_wallet#3574](https://github.com/cake-tech/cake_wallet/issues/3574) persist still lags one event). [kiss-bdk](https://github.com/kkdao/kiss-bdk) today treats `done` as the asked range finished ([kiss-bdk#10](https://github.com/kkdao/kiss-bdk/issues/10)). Each height carries tweak + txid + taproot `output_pubkeys` (client scans locally; no block fetch). Naive walk, or `--sptweaks` thin index (`len:tweak` only; one `txout` span per wave; the first height shares that span when indexed). Pre-taproot: **one** notify with ≤1024 empty height keys (Cake `fromJson` last key is progress); probe `[0,1,false]` stays `{"0": {}}`. Param `[2]` (Cake `historicalMode`): `false` **cut-through** (omit confirmed-spent P2TR outs / txs); `true` keeps spent outs. Serve-time `--sptweaks-dust` (default **1000**) omits P2TR outs with `value <=` the floor (`0` = all; **546** matches Cake electrs `sp_min_dust`). Sparrow Silent Payments uses Frigate `blockchain.silentpayments.subscribe` (server-side scan) — **not** this method. Cake isolate may still hardcode `electrs.cakewallet.com`. | +| blockchain.tweaks.subscribe | done | **Stream**, not a one-shot result: JSON-RPC `result` is the **first** height only; remaining heights are unsolicited notifications; `{"message":"done"}` ends the **chunk** (60s wall at a wave boundary, or the requested `count` if sooner) — Cake electrs clamps `count` to 1000 instead. Cake Wallet resubscribes from `syncHeight+1` ([cake_wallet#3574](https://github.com/cake-tech/cake_wallet/issues/3574) persist still lags one event). [kiss-bdk](https://github.com/kkdao/kiss-bdk) today treats `done` as the asked range finished ([kiss-bdk#10](https://github.com/kkdao/kiss-bdk/issues/10)). Each height carries tweak + txid + taproot `output_pubkeys` (client scans locally; no block fetch). Naive walk, or `--sp-tweaks` thin index (`len:tweak` only; one `txout` span per wave; the first height shares that span when indexed). Pre-taproot: **one** notify with ≤1024 empty height keys (Cake `fromJson` last key is progress); probe `[0,1,false]` stays `{"0": {}}`. Param `[2]` (Cake `historicalMode`): `false` **cut-through** (omit confirmed-spent P2TR outs / txs); `true` keeps spent outs. Serve-time `--sp-tweaks-dust` (default **1000**) omits P2TR outs with `value <=` the floor (`0` = all; **546** matches Cake electrs `sp_min_dust`). Sparrow Silent Payments uses Frigate `blockchain.silentpayments.subscribe` (server-side scan) — **not** this method. Cake isolate may still hardcode `electrs.cakewallet.com`. | | headers / block headers | done | Tip push on subscribe | | scripthash history / balance / listunspent | done | Unconf when mempool attached; `get_history` optional BCH-style `from_height` / exclusive `to_height` (`-1` = tip + mempool); 1-arg = full history; unconfirmed `get_history` rows include `fee` (same as `get_mempool`); `listunspent` mempool height is `0` or `-1` (unconfirmed parent); **subscribe status always full**; `blockchain.scripthash.unsubscribe` returns whether the connection was watching (frees a slot under `--electrum-max-subs`, default 10000 per connection). `listunspent` loads `txid.body` only for unspent creates; one TCP connection reuses the last SH outs+spent join until SH-view **hash** changes. Confirmed methods stamp live tip: a RAM SH head (pending write-behind) joins with durable SH so mempool can drop confirmed txs without a hole. Durable Class B seed waits until after tip announce. `get_history` skips mempool rows already in the confirmed list. `server.features.chain_tip = true`. Trailing **`asof:`** after the official args (`server.features.asof` / `asof_protocol = 1.4.2-asof`): confirmed rows as of that still-live ancestor **at or behind visible SH** (durable + pending), **no** mempool; stamp is the asof block; unknown hash or ahead of visible SH → `asof not on chain`. Prefix keeps it off the future positional-string landmine. Requires negotiated `1.4.2-asof` (first `server.version` only). Electrum `protocol_max` is **1.6**. | | scripthash.get_mempool / subscribe | done | Status on mempool announce **and** when SH applies a height that creates or spends the hash (posting-list probe; no Class A expand on a miss). Headers subscribe still live tip. Reorg (`TipNotify.reorg_from_height`) restatuses every watch even if the new block misses the script. Status preimage is `txid:height:blockhash:` for confirmed rows (mempool rows stay `txid:height:`). Row **order** is confirmed height-asc then mempool tail, same as `get_history`. A script with no history has status `null` (subscribe reply and pushes), never `""`: wallets read any non-null status as a used address. `unsubscribe` is implemented. | | transaction.get / get_merkle | done | get falls back to mempool unless `asof:`; confirmed responses stamp `chain_tip`. Verbose get is electrs-shaped: `time`/`blocktime`/`confirmations`/`blockhash` from the confirming header (mempool verbose is `confirmations: 0` with no block stamp), plus `vin`/`vout`/`size`/`version`/`locktime`/`hash` (wtxid) decoded from the hex already loaded. Trailing `asof:` (same dialect as scripthash): get returns the tx only if confirmed at or behind that ancestor (**no** mempool); get_merkle rejects a `height` above the asof pin (`asof not on chain`). `id_from_pos` is a txid string; third arg `merkle=true` returns `{tx_hash, merkle}` (same walk as `get_merkle`). | | transaction.broadcast | done | Mempool accept + P2P inv. `broadcast_package` wraps `accept_package` (Electrum **1.6**; P2P BIP331 still Q-48). | -| relayfee / estimatefee / histogram / `mempool.get_info` | done | `minrelaytxfee` is the configured minimum. `relayfee` and `mempoolminfee` are the enforced floor. `incrementalrelayfee` stays 100 sat/kvB. Estimates still use the live median. `mempool.get_info` is 1.6 (`minrelaytxfee` replaces `relayfee` for 1.6 clients; `relayfee` stays for 1.4). | +| relayfee / estimatefee / histogram / `mempool.get_info` | done | `minrelaytxfee` is the configured minimum. `relayfee` and `mempoolminfee` are the enforced floor. `incrementalrelayfee` stays 100 sat/kvB. Estimates are the 10-minute flow/history product ([`docs/mempool-fee-estimation.md`](./docs/mempool-fee-estimation.md)). `mempool.get_info` is 1.6 (`minrelaytxfee` replaces `relayfee` for 1.6 clients; `relayfee` stays for 1.4). | | outpoint.get_status / subscribe / unsubscribe | done | Electrum **1.7** methods; `protocol_max` stays **1.6** until `scriptpubkey.*`. Spent = confirmed-strong or mempool. | | silentpayments.subscribe / unsubscribe | done | Frigate remote-scanner: session-only scan key; historical + tip notifies via tweak index / naive `tweaks_for_height`. Not Cake `tweaks.subscribe`. | | TLS | external | terminate at reverse proxy for **clearnet**; onion is plain TCP (`ADD_ONION` to loopback). In-binary 50002 + TLS is parked **Q-63**. | @@ -210,9 +210,9 @@ hash. Electrum clients that want as-of send `server.version(name, "1.4.2-asof")` (or a `[min, max]` range containing that string). Standard Electrum `"1.4"` / `["1.4", "1.4.2"]` stays on dotted-int 1.4.x; an `asof:` tag -without the dialect is an error. `server.features.protocol_max` remains -`"1.4.2"` so Electrum dotted-int parsers do not choke; discovery is -`asof` + `asof_protocol`. +without the dialect is an error. `server.features.protocol_max` stays +`"1.6"`. As-of discovery is `asof` / `asof_protocol` (`1.4.2-asof`), not +`protocol_max`. ## Esplora REST surface diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c1db59cf6..5f57ce866 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -269,7 +269,7 @@ does not apply to them. Humans who want the same gates offline (Nix optional; rustup 1.95 is enough): ```bash -# nix develop # Linux only — pin via flake.lock; or rustup + rust-toolchain.toml +# nix develop # Linux and aarch64-darwin — pin via flake.lock; or rustup + rust-toolchain.toml export CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$PWD/target/dev}" cargo fmt --all -- --check # includes nested fuzz/ crate # rustc warnings are denied via workspace.lints (+ RUSTFLAGS=-Dwarnings in the Nix shell) diff --git a/SCHEMA.md b/SCHEMA.md index c3c792763..66aa456b7 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -101,8 +101,9 @@ Older versions and migration notes live in [`SCHEMA_HISTORY.md`](./SCHEMA_HISTOR ## Schema 17 freeze Class A shape settled here at schema 17 and is still the live layout under -the `SCHEMA_VERSION` at the top of this file (25). Schemas 18–24 already -shipped; they are open rules above and history in +the `SCHEMA_VERSION` at the top of this file (26). Schemas 18–26 already +shipped. Meta 22–26 is what this binary opens (rules above). Earlier +layouts are history in [`SCHEMA_HISTORY.md`](./SCHEMA_HISTORY.md). A byte-incompatible change bumps from that live constant ([Changing durable bytes](#changing-durable-bytes)). It does not target @@ -230,7 +231,7 @@ itself changed. tx.body / tx.idx.* # schema ≤14 packed (refused if non-empty) txid.body # dense create_fk-ordered txids (schema 13+) txstat.body / txstat.ovf / txstat.blk # 8 B/create ULEB econ + per-header tails (schema 25) - input.loc / input.off / input.body # spender → parent edges (schema 22–23) + input.loc / input.off / input.body # spender → parent edges (8 B/input) + n_in (2 B/create) tx.head/ # meta + open OA NNNNNN; sealed NNNNNN.mphf|.fuse8 spent.ovf # multi-spender overflow (was spenders.body) confirmed.body # Class C: height → header_fk @@ -284,7 +285,7 @@ store. `txstat`, `input`, block filters, and tweaks are not pruned. | Offset | Size | Field | |--------|------|-------| | 0 | 4 | Magic `RBT1` | -| 4 | 2 | Schema version (u16) — live **25** (`SCHEMA_VERSION`). Occupied files keep the version they were written; **13–25** remain `schema_file_openable` | +| 4 | 2 | Schema version (u16) — live **26** (`SCHEMA_VERSION`). Occupied files keep the version they were written; **22–26** remain `schema_file_openable` | | 6 | 2 | Table kind (u16) | | 8 | 8 | Logical length (bytes), including this header | @@ -314,6 +315,7 @@ store. `txstat`, `input`, block filters, and tweaks are not pruned. | 21 | txstat per-header locator (`txstat.blk`, 16 B/header) | | 22 | inputs (`input.body`, 8 B/input) | | 23 | inputs loc (`input.loc`, 2 B/create `n_in`) | +| 24 | blockfilter (`blockfilter.body`) | --- @@ -609,7 +611,7 @@ Contiguous assignment required: block membership is an arithmetic range. ### Optional BIP-352 thin tweaks (`sp_tweaks.*`) Schema **17** side product. Soft-open: missing dirs are empty (not `Corrupt`, -not a head recreate). Created when `--sptweaks` is on. +not a head recreate). Created when `--sp-tweaks` is on. **Tip / strong height only** — no `header_fk` in the idx. A reorg truncates above the new tip and those heights are written again. `put` requires @@ -629,7 +631,7 @@ slot in that file, or that file’s HWM). `n_tx` comes from Leftover **files** `store/sp_tweaks.idx` and `store/sp_tweaks.body` (schema 14 single-file, `header_fk` + absolute off) are unlinked on store open. -`--sptweaks` backfill regenerates. Not a Class A wipe. +`--sp-tweaks` backfill regenerates. Not a Class A wipe. Reorg: truncate slots above the new tip (same era as SH HWM). @@ -860,9 +862,10 @@ Schema 15 leftover `tx_height.body` is unlinked on open (logged). ### Commit order (confirm) -1. `strong_tx` (may lead tip after kill) -2. Thin scripthash creates (may lead tip) -3. **`confirmed[]` tip advance** ← **commit**, then fence extend +1. `strong_tx` (may lead tip after kill, before the barrier) +2. Height-fence extend (RAM), then **`confirmed[]` tip advance** ← **commit** +3. `flush_class_c_tip` (headers, then `strong_tx`, `header_txs`, `confirmed[]` last) +4. Scripthash write-behind enqueue (after tip). Direct IBD defers SH until tip entry `is_confirmed_strong(tx)` ⇔ strong ∧ fence contains the fk (implies height ≤ tip and membership in `confirmed[h]` header_txs). diff --git a/SCHEMA_HISTORY.md b/SCHEMA_HISTORY.md index aae9e54ae..b4617c286 100644 --- a/SCHEMA_HISTORY.md +++ b/SCHEMA_HISTORY.md @@ -126,7 +126,7 @@ main/L1 writes `BDZ3`: 3-partite peel, 2-bit `g`, occupancy rank (RAM on open), then mix64 tags + pack8 `.val`. Occupied schema 18/19 `tx.head` or `scripthash*` is refused (wipe those dirs, keep Class A). Empty indexes rewrite `meta` to 22; `tx.head` rebuilds from `txid.body`; SH rematerializes -with `--shindex`. Open OA is still 4 B rel. +with `--sh-index`. Open OA is still 4 B rel. ## v19 (megakey extent) @@ -178,7 +178,7 @@ since shipped. single-file `sp_tweaks.idx` / `sp_tweaks.body` (schema 17 tweaks are `sp_tweaks.idx/` + `sp_tweaks.body/` directories: tip-only `off:u32` slots, original `0`/`33` body, new segment when the next start would -exceed `u32::MAX`). `--sptweaks` backfill regenerates the index. +exceed `u32::MAX`). `--sp-tweaks` backfill regenerates the index. ### Side product: `sp_tweaks.*` (schema 17 dirs) @@ -360,8 +360,9 @@ log a one-line warn, migrate or refuse with a clear message — do not silently - Rehash at load **7/8** (earlier eras rehashed more aggressively, e.g. ~1/2). Current writer: insert past 7/8 is full; `header.head` rolls `header.head.gN` instead of rewriting occupied slots. Open-grow of an undersized single gen - deletes the OA file and recreates it at the create target (exclusive, no - concurrent probes; `.mlt` kept). + writes `header.head.grow`, fsyncs, then renames over the live file + (exclusive, no concurrent probes; `.mlt` kept). A crash during the rewrite + leaves the previous OA. ### Why diff --git a/TESTING.md b/TESTING.md index e634a1843..df4f7622c 100644 --- a/TESTING.md +++ b/TESTING.md @@ -168,7 +168,7 @@ absolute path. Keep default-tier fixtures small: on tmpfs, test bytes are RAM. **Baseline (agent VM, warm test profile, 2026-08-07):** full `cargo test --workspace` was **~1000 s (~17 min)** before store fan-in scale fixes. After parameterizing fan-in targets and shrinking SH head default benches (`6588b62` era): `rbitcoin-store --lib` serial **~26 s** (was **~498 s**); `sorted_run` module **~1 s** (was **~191 s**). -**CI-class (2026-08-17):** required GitHub Actions `test` job (`ubuntu-24.04`, `cargo test --workspace` + node/cli build) is **~85 s** (PR 85). That meets the ≤3 min budget and the **<2 min** stretch on CI hardware. Do **not** re-run multi-minute full-suite timing loops as a planning spike; package walls below are still the local budget if a change feels slow. +**CI-class (2026-08-17):** required GitHub Actions `test` job (`ubuntu-latest`, `cargo test --workspace` + node/cli build) is **~85 s** (PR 85). That meets the ≤3 min budget and the **<2 min** stretch on CI hardware. Do **not** re-run multi-minute full-suite timing loops as a planning spike; package walls below are still the local budget if a change feels slow. **Agent VM (2026-10-03, warm test profile, tmpfs runner):** `cargo test --workspace` was **199.3 s** on `1d2f8896` and **117.2 s** / **118.5 s** after this cut, rebased onto current master. `end_of_ibd` went from **83.8 s** to **~23 s** (the work-fork journey was two 30 s stall waits, then **~5 s**). `cross_surface` went from **19.3 s** to **~9.5 s**. `integration_multinode` stayed **~32 s**. @@ -252,7 +252,7 @@ All workspace members that contain production code: - `rbitcoin-primitives`, `rbitcoin-store`, `rbitcoin-query` - `rbitcoin-consensus`, `rbitcoin-mempool`, `rbitcoin-net` - `rbitcoin-electrum`, `rbitcoin-esplora`, `rbitcoin-log` -- `rbitcoin-rpc`, `rbitcoin-cli`, `rbitcoin-node` +- `rbitcoin-rpc`, `rbitcoin-cli`, `rbitcoin-node`, `rbitcoin-sv2` **Excluded by default:** third-party crates, `src/main.rs` trampolines, test modules (`*_tests.rs`, `tests.rs`, crate `/tests/`, `testutil.rs`, @@ -465,7 +465,7 @@ API. ./scripts/fuzz-run.sh v2_session # live Core v2 peer, ping/pong compare, ASan ./scripts/fuzz-run.sh cmpct_differential # compact missing indexes vs getblocktxn, ASan ./scripts/fuzz-run.sh block_differential # fetch bitcoind, --sanitizer none -./scripts/fuzz-run.sh block_spend_differential # 100-block pad; weekdays --sanitizer none -timeout=180; Sunday address -timeout=30 +./scripts/fuzz-run.sh block_spend_differential # 100-block pad; weekdays --sanitizer none -timeout=180; Sunday address -timeout=90 ./scripts/fuzz-run.sh script_differential # mutate executed scriptPubKey, --sanitizer none ./scripts/fuzz-run.sh block_fork_differential # pad+stem, 2-block fork, --sanitizer none ./scripts/fuzz-run.sh cmpct_reorg_differential # child-first drain_pending vs Core diff --git a/changelog.d/docs-match-code.md b/changelog.d/docs-match-code.md new file mode 100644 index 000000000..5eca4e1a6 --- /dev/null +++ b/changelog.d/docs-match-code.md @@ -0,0 +1,16 @@ +Fixed + +- **Operator and schema docs match the node.** Blocks in transit per peer + are 64. `--mempool-size-mb` is N × 1_000_000 weight units. Scripthash + extract workers are one per 1.5 GiB. Confirm commits the height fence + and `confirmed[]` before scripthash write-behind. Schema file headers + are version 26, including block-filter kind 24 and `input.*`. +- **IBD and mempool operator notes match the code.** Densify is 32 + hashes per peer (64 for a fast outlier). Cluster caps are 64 txs and + 101 kvB. The mempool sidecar is schema 3. `tip: accept` is DEBUG. + Linux falls back to pread when io_uring cannot open. +- **CLI names in the product docs are `--sh-index` and `--sp-tweaks`.** + Electrum and Esplora start with the scripthash index off. `generate*` + also runs on an `OP_TRUE` signet. Electrum `protocol_max` is 1.6. + SV2 template-provider plans A–C are landed. CLN and ldk-node chain + backends are the contract in `lightning.md` (Q-69 closed). diff --git a/crates/rbitcoin-consensus/AGENTS.md b/crates/rbitcoin-consensus/AGENTS.md index 962effda8..7e6da2c2a 100644 --- a/crates/rbitcoin-consensus/AGENTS.md +++ b/crates/rbitcoin-consensus/AGENTS.md @@ -17,8 +17,8 @@ Open the row that matches the change. Leave the other owners closed. ## Where - Scripts: `src/script/` (`interpreter.rs` is the opcode match) -- Confirm: `src/confirm_run.rs` -- Headers and blocks: `src/header.rs`, `src/block.rs` +- Confirm: `src/confirm_run/` +- Headers and blocks: `src/header.rs`, `src/block/` - Worker pool: `src/script_pool.rs` ## Verify diff --git a/crates/rbitcoin-esplora/AGENTS.md b/crates/rbitcoin-esplora/AGENTS.md index a966033f1..6230c3f2c 100644 --- a/crates/rbitcoin-esplora/AGENTS.md +++ b/crates/rbitcoin-esplora/AGENTS.md @@ -1,8 +1,8 @@ # rbitcoin-esplora -Esplora-compatible REST. Depends on query, store, -net, and mempool. Electrum is a sibling crate (`rbitcoin-electrum`); both -need `--sh-index`. +Esplora-compatible REST. Depends on query, store, electrum, +net, and consensus. Electrum is a sibling crate (`rbitcoin-electrum`); both +need `--sh-index` for address history. ## Read first diff --git a/crates/rbitcoin-node/src/cli.rs b/crates/rbitcoin-node/src/cli.rs index b1b31fe78..e8dc2b843 100644 --- a/crates/rbitcoin-node/src/cli.rs +++ b/crates/rbitcoin-node/src/cli.rs @@ -336,7 +336,7 @@ Milestone: skip script/sig checks at/below HEIGHT.\n\ (skip only on that header path, and only when header work meets that network's min chain work),\n\ signet 0, regtest 0. Explicit HEIGHT is height-only. Use 0 for full scripts.\n\ Check-blocks: --check-blocks N revalidates the last N confirmed heights on open (default 6; 0 = all).\n\ -Mempool: --mempool-size-mb (default ~300 MiB weight budget).\n\ +Mempool: --mempool-size-mb (default 300 → 300e6 WU; N × 1_000_000 weight, not MiB).\n\ Peers: --max-outbound (default 16 live download), --max-inbound (default 125).\n\ --proxy HOST:PORT SOCKS5 for all P2P outbound; --onion HOST:PORT SOCKS for onion (02).\n\ --proxy-randomize (default on) uses a fresh SOCKS username per peer (Tor circuit isolation).\n\ diff --git a/crates/rbitcoin-rpc/AGENTS.md b/crates/rbitcoin-rpc/AGENTS.md index a1cd65a19..032aa5dd8 100644 --- a/crates/rbitcoin-rpc/AGENTS.md +++ b/crates/rbitcoin-rpc/AGENTS.md @@ -2,7 +2,7 @@ Core-class JSON-RPC subset. Not full Core parity. Depends on query, store, net, and consensus. The HTTP server is `src/server.rs`; methods are -`src/methods.rs`. +`src/methods/`. ## Read first diff --git a/docs/architecture.md b/docs/architecture.md index 686613ab3..698917c6d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -50,7 +50,7 @@ for unknown-height bodies (mark missing → re-getdata). crypto primitive via the rust-bitcoin stack — **no** `libbitcoinconsensus` dual-eval). - **Electrum** is **native** to the store (optional **scripthash** tables via - `--shindex`, default off), not a second process re-indexing blk files. + `--sh-index`, default off), not a second process re-indexing blk files. Tip-follow does **not** wait on SH materialize; Electrum/Esplora do. SH tip materialize is unsorted per-shard files then unique-sort; cold megakey pages stream into 4 KiB delta pages (schema 17). diff --git a/docs/concurrency.md b/docs/concurrency.md index dcbe841d6..17e20dd1c 100644 --- a/docs/concurrency.md +++ b/docs/concurrency.md @@ -15,7 +15,7 @@ Short map of who may write which tables. **Format is unstable until 1.0.** **IoSession TLS:** one completion session per OS thread (`with_thread_local`). Harvest / poison / drain / do-not-flatten: [`io-modality.md`](./io-modality.md). `RBITCOIN_IO=pread` disables the session. SH k-way merge submits 256 KiB ahead preads on that TLS session and waits only when promote needs a page that has not completed. -**Height-ordered unified pipeline (current):** peer → **body queue** (raw only) → **lookup** (in-order from `max(path_lo, lookup_taken_hi+1)`; decode + TipOnly `head_fk`; **dequeue** raw into `loadq=14`). Hole/densify/receive: in-hand = confirmed ∨ BQ hash ∨ `H ≤ lookup_taken_hi` (Cancelled leaves taken_hi; every other post-lookup reject re-arms it at the confirmed tip; a load stamp, scripts, or write EngineFault re-offers the **full** batch including the first height after that re-arm and does not `request_single_block`; a load pin EngineFault retries one block at a time, as a cascade, since it can be a stale-fk plan race; a load reject bumps the feed epoch, so valid lower batches already in scripts or write are dropped and their bodies fetched again from peers — the price of a consistent restamp; epoch-stale loadq chunks re-offer on drop; BadPrev most-work rewind then sets the LCA) → **load** (recv load-sized batch; stamp binds carried BQ keys + BQ `header_fk` (skip header ensure/put when the store row matches) + pin + assemble) → scripts → write. Stage IO: [`invariants.md`](./invariants.md). **No** peer→confirm-feed wire retain. **No** hash-only / Class-A-only confirm (bq wire required). Load bind order is same-batch → in-flight → load-batch skeleton → Corrupt (plan=None leftover TipOnly). Lookup does not own a published identity chain. Class A sets loc once on the same `CreatePin` Arc load cloned into InFlight (`OnceLock`); later-wave stamp reads it (no disk loc-by-fk on that path). Confirm writes header size/weight as a sequential `header.body` rewrite of 96-byte records (`put_size_weight_run`, no insert `put_lock`); header-sync insert still stores zeros. Write drain inserts `tx.head` in parallel with Class C on the process-wide `ibd-confirm-head` thread. Drain complete is max inserted **fk** (not tip/fence — those advance during drain). Header-cache GC polls store tip every load pack. In-flight prune is **after the last load batch of a lookup wave finishes its in-flight read**, using the drain+fence height snapshotted before that wave's TipOnly (`docs/invariants.md`). No leftover pending map. Bodies without a known height are marked missing and re-getdata after the height map is ready — there is **no** dual-track archive-job / ContigPark fallback. Load pack **waits** on `feed.cv` when tip+1 is in `ready` but the BQ is not resolve-complete (no retain/BQ spin). Pack takes the feed mutex to collect candidates and again to mark inflight; one BQ `pack_snapshot` in between. +**Height-ordered unified pipeline (current):** peer → **body queue** (raw only) → **lookup** (in-order from `max(path_lo, lookup_taken_hi+1)`; decode + TipOnly `head_fk`; **dequeue** raw into `loadq=14`). Hole/densify/receive: in-hand = confirmed ∨ BQ hash ∨ `H ≤ lookup_taken_hi` (Cancelled leaves taken_hi; every other post-lookup reject re-arms it at the confirmed tip; a load stamp, scripts, or write EngineFault re-offers the **full** batch including the first height after that re-arm and does not `request_single_block`; a load pin EngineFault retries one block at a time, as a cascade, since it can be a stale-fk plan race; a load reject bumps the feed epoch, so valid lower batches already in scripts or write are dropped and their bodies fetched again from peers — the price of a consistent restamp; epoch-stale loadq chunks re-offer on drop; BadPrev most-work rewind then sets the LCA) → **load** (recv load-sized batch; stamp binds carried BQ keys + BQ `header_fk` (skip header ensure/put when the store row matches) + pin + assemble) → scripts → write. Stage IO: [`invariants.md`](./invariants.md). **No** peer→confirm-feed wire retain. **No** hash-only / Class-A-only confirm (bq wire required). Load bind order is same-batch → in-flight → load-batch skeleton → Corrupt (plan=None leftover TipOnly). Lookup does not own a published identity chain. Class A sets loc once on the same `CreatePin` Arc load cloned into InFlight (`OnceLock`); later-wave stamp reads it (no disk loc-by-fk on that path). Header records are 88 B consensus fields. Block size and weight are stamped on `txstat`, not rewritten into `header.body`. Write drain inserts `tx.head` in parallel with Class C on the process-wide `ibd-confirm-head` thread. Drain complete is max inserted **fk** (not tip/fence — those advance during drain). Header-cache GC polls store tip every load pack. In-flight prune is **after the last load batch of a lookup wave finishes its in-flight read**, using the drain+fence height snapshotted before that wave's TipOnly (`docs/invariants.md`). No leftover pending map. Bodies without a known height are marked missing and re-getdata after the height map is ready — there is **no** dual-track archive-job / ContigPark fallback. Load pack **waits** on `feed.cv` when tip+1 is in `ready` but the BQ is not resolve-complete (no retain/BQ spin). Pack takes the feed mutex to collect candidates and again to mark inflight; one BQ `pack_snapshot` in between. **Load claim pack size:** soft **Σ `tx.input`** budget (hardcoded **8000**; include overshoot block) or hard **144** blocks. Also stop a `LoadBatch` before the next height when `header_txs.has_body` differs from the part's first height (crash some→none). Lookup may still decode a mixed resolve wave; loadq chunks are one kind. Dense mainnet blocks hit the input soft stop after **typically a few blocks** (often 1–3); early tiny blocks may pack many until the hard cap. Do **not** treat ~32 as pack size (that was 8000/250 mid-chain, not fat-era). diff --git a/docs/crash-recovery.md b/docs/crash-recovery.md index cce2a4cd2..169fa9c05 100644 --- a/docs/crash-recovery.md +++ b/docs/crash-recovery.md @@ -14,9 +14,9 @@ Best-chain views ignore uncommitted Class C state: |-------------------------------------------|------| | 0. Structural spentness / maturity / subsidy | No durable tip write yet | | 1. `strong_tx` (L2 RAM) | May lead tip after kill **before** barrier | -| 2. Thin scripthash **creates** (batched) | May lead tip after kill | -| 3. `confirmed[]` tip advance (L2 RAM) + height-fence extend | In-process commit | -| 3b. **`flush_class_c_tip`** (complete-or-fail L2 images) | **Durability barrier** | +| 2. Height-fence extend (RAM), then `confirmed[]` tip advance (L2 RAM) | In-process commit | +| 3. **`flush_class_c_tip`** (headers, then `strong_tx`, `header_txs`, `confirmed[]` last) | **Durability barrier** | +| 3b. Scripthash write-behind enqueue | After tip. Direct IBD defers SH until tip entry | | 4. Body-queue dequeue for those heights | Only after confirm-write returns Ok | | 5. Spend annotations (Direct) | After tip. A missing slot is unspent until open replays from the marker | @@ -68,11 +68,11 @@ Open revalidation runs in `Query::open_or_create` **before** P2P can extend tip. ## Thin scripthash (Electrum outpoint pointers) -- Schema 15: head holds ≤2 inline creates, a geometric **slab** (3–256 fks, ULEB128 +- Schema 18: the head value holds **one** inline create (`SH_INLINE_CAP`), a geometric **slab** (3–256 fks, ULEB128 deltas), or a megakey **page chain** (≥257). Size-class freelist reuses freed slabs. Main heads are sealed MPHF+pack8 (no fuse); new keys go to global ingest OA. -- **No spend columns** — spentness from points + Class C at query time. -- Creates written on confirm (before tip advance). +- **No spend columns** — spentness is the `spent.body` annotation plus `is_confirmed_strong`. +- Direct IBD does not enqueue SH creates. Tip write-behind runs after `flush_class_c_tip`. Tip entry bulk-materializes from Class A. - **Kill-safe without chain walks:** first confirm after open **sequentially scans** `scripthash.body` once into a process set of `create_tx_fk`s already present; re-confirm skips those txs. Hot path only appends + maintains heads in RAM. @@ -87,7 +87,7 @@ Clean shutdown: `flush_for_shutdown` fsyncs tip/Class C (incl. L2 dirty images) Steady path: payload pwrite + HWM publish; `sync_data` on flush barriers. Kill mid-payload before HWM publish: readers never see past previous published length. -Connect barrier (`flush_class_c_tip`): headers (if dirty) → strong → height → header_txs → **confirmed last** → soft `tip_seal` (tmp + `sync_all` + rename + parent-directory `fsync`). The barrier does not `sync_data` Class A. +Connect barrier (`flush_class_c_tip`): headers (if dirty) → strong → header_txs → **confirmed last** → soft `tip_seal` (tmp + `sync_all` + rename + parent-directory `fsync`). The height fence is RAM-only and is not flushed. The barrier does not `sync_data` Class A. Disconnect: confirmed truncate + `flush_confirmed_only` (also refreshes `tip_seal`) before unstrong/height clear. If the new tip is below `spend_durable`, that marker is lowered to the tip. Confirm stores a snapshot height after annotations are written and does not `sync_data` these stems. `rbtc-spend-sync` does, about every 10 minutes: `sync_data` on the replay stems and `spenders` (not a high-water rewrite, not `tx.head`), then if the confirmed tip is still at least that snapshot, publish `A = D` at the snapshot. Blocks appended above the snapshot during the barrier are flushed along and are not claimed. A disconnect bumps an in-process generation before it lowers the snapshot and the marker, including when no marker file exists yet. The checkpoint samples that generation before `sync_data` and again under the marker lock; it publishes only when the generation is unchanged, the tip is still at least the snapshot, and no annotate is pending. A disconnect and a reconnect of a different block at the snapshot height during that window leave the previous marker in place. An append strictly above the snapshot does not bump the generation, so the marker still publishes. One still pending caps `A` below its first height. A failed marker publish leaves the previous marker and does not start a new interval. Shutdown wakes the thread for one checkpoint, joins it, then runs `flush_for_shutdown`. The barrier is `File::sync_data` (Linux `fdatasync`, macOS `F_FULLFSYNC`, Windows `FlushFileBuffers`). It is not inside `flush_class_c_tip`, and it is not charged on the confirm thread's `ann_sync` timer. A cookie inside a data file, or a clean page cache, is not this cursor: writeback is not a device flush, and the kernel may reorder data ahead of a later header write. @@ -116,11 +116,11 @@ Private, **not** Class A. RAM graph is source of truth; files may lag. ## Operator - Direct IBD keeps segmented **`tx.head/`** (archive) and **spend annotations** (confirm) live; tip entry does **not** re-scan Class A to repair them. Corrupt head/spends ⇒ reindex (optional manual `backfill_tx_index` rebuilds segmented head mappings from Class A). -- **Segmented `tx.head`:** directory `tx.head/` with `meta` + open OA `NNNNNN`; sealed `NNNNNN.mphf` + `.fuse8`. Packed BDZ `g` is FdOnly (4 KiB page stream); MPHF output is `rel−1`. Flat `tx.head.meta` / `tx.head.NNNNNN` are **migrated into** `tx.head/` on open. Roll opens the next OA first; seal runs on a sidecar and publishes later. Seal/install of `meta` / `.mphf` is sibling tmp + `sync_all` then rename (empty truncate of the live name is not a seal). Publish persists sealed `meta` **before** unlinking the segment's OA; a leftover OA next to a sealed `.mphf` is discarded on open. Kill mid-seal leaves **at most one** unsealed non-tail OA: open collects fuse keys once from `txid.body` and seals it (does not retain `open_keys`). Two unsealed non-tails is **Corrupt**. Leftover fuse8 v1 and flat `tx.head.meta` **refuse** (`Query::open`); wipe `store/tx.head` (Class A kept) then restart. Unreadable **current** head (empty/truncated `meta`, truncated/missing sealed MPHF) with Class A: wipe+rebuild from `txid.body` (same cost as a clean wipe). Wipe or empty occupancy + Class A: open rebuilds **MPHF+fuse8 directly** from `txid.body` in parallel (default **2²⁵** keys/range; `RBITCOIN_TX_HEAD_REBUILD_WORKERS`); no historical OA. Legacy mono `tx.head` file / `.new` / `.resize` are not opened — reindex. +- **Segmented `tx.head`:** directory `tx.head/` with `meta` + open OA `NNNNNN`; sealed `NNNNNN.mphf` + `.fuse8`. Packed BDZ `g` is FdOnly (4 KiB page stream); MPHF output is `rel−1`. Flat `tx.head.meta` **refuses** on open (wipe `store/tx.head`; Class A kept). Roll opens the next OA first; seal runs on a sidecar and publishes later. Seal/install of `meta` / `.mphf` is sibling tmp + `sync_all` then rename (empty truncate of the live name is not a seal). Publish persists sealed `meta` **before** unlinking the segment's OA; a leftover OA next to a sealed `.mphf` is discarded on open. Kill mid-seal leaves **at most one** unsealed non-tail OA: open collects fuse keys once from `txid.body` and seals it (does not retain `open_keys`). Two unsealed non-tails is **Corrupt**. Leftover fuse8 v1 and flat `tx.head.meta` **refuse** (`Query::open`); wipe `store/tx.head` (Class A kept) then restart. Unreadable **current** head (empty/truncated `meta`, truncated/missing sealed MPHF) with Class A: wipe+rebuild from `txid.body` (same cost as a clean wipe). Wipe or empty occupancy + Class A: open rebuilds **MPHF+fuse8 directly** from `txid.body` in parallel (default **2²⁵** keys/range; `RBITCOIN_TX_HEAD_REBUILD_WORKERS`); no historical OA. Legacy mono `tx.head` file / `.new` / `.resize` are not opened — reindex. - Scripthash: Direct IBD **defers** SH (no memtable, no confirm enqueue). After the horizon, two Class A `txout` scans: each worker owns a contiguous create-fk span and unsized maps (1.5 GiB estimate cap; spill the largest shard map while over budget; one writer, 1-slot queue) into `scripthash.unsorted/keys/NN/` (`SHKSP01` files, first-fk delta singles; tmp+rename, not a kill-9 barrier — no `DONE.keys` still wipes unsorted); merge folds those spills into one map, one walk to `scripthash.head/NN` + `multi/NN.fuse8`, and unlinks `keys/NN/`; then fuse-hit `post` (`SHPST01` spills under `post/NN/`), pack folds those spills then 2+ bodies, unlink each shard's extract as it seals. A **durable head** on restart stays Tip: write-behind / `recover_sh_writebehind` fills any HWM lag; leftover `scripthash.runs` are discarded (not WarmOnly-merged). - **Full cold** when head empty. **`RBITCOIN_SH_FORCE_REBUILD=1`:** wipe head + full two-scan collect + pack. Empty collect after Class A creates remain is fatal. - **Cold resume:** the pack commit is `scripthash.head/NN.packed` (sibling of the MPHF base; holes stay). Open loads a shard's MPHF only when that mark exists. `MphfHead::exists` after pass-1 BDZ is **not** pack-done. A complete unmarked head (every shard has `.mphf`+`.val`, and `scripthash.unsorted` has no `DONE.keys` / `DONE.post` / `keys/` / `post/`) is soft-migrated: the marks are written, and a missing `include_hwm` is set from the create count. An extract still on disk never gets a mark, even if `include_hwm` already names the tip. No valid `DONE.keys` (including leftover `SHUNSRT3` / 24 B `NN`, or `keys/NN` / `post/NN` as a file) deletes unsorted and restarts pass 1. A spill whose first 8 B are not `SHKSP01` is Corrupt — wipe unsorted; do not parse or migrate. `DONE.keys` / `DONE.post` record the inclusive Class A `create_fk` scanned (`SHKEYS02` last_fk marker / `SHPOST02`). Missing current `keys/NN/` spills with unsealed shards finishes BDZ from those files. Head files present and no `DONE.post` restarts pass 2 from fk 1 (partial `post/` spills are deleted and recollected). Packing a shard soft-clears ingest keys that packed main now owns, so a tip append that landed on ingest while the shard was unsealed cannot hide that chain. A `post/NN` **file**, or a spill whose first 8 B are not `SHPST01`, is Corrupt (wipe unsorted). If Class A grew after `DONE.post` and **no** shards are packed, append postings. If any shard is already pack-marked, pack remaining unsealed `post/NN/`, then Class A tail-append onto the durable head (Direct write-behind no-ops). After all shards seal, the unsorted dir is removed. - Empty head + leftover catalog: wipe leftover runs + SEAL, then Class A collect (not k-way from `scripthash.runs`). Durable head: leftover runs are discarded, `SEAL` kept; missing `include_hwm` bootstraps from SEAL. Inclusion HWM: `scripthash.include_hwm`. **Leftover live OA** at `scripthash.head` (or non-`SHSR` `ovf/NNNNNN`): refuse — wipe `store/scripthash*` and restart with `--shindex`. + Empty head + leftover catalog: wipe leftover runs + SEAL, then Class A collect (not k-way from `scripthash.runs`). Durable head: leftover runs are discarded, `SEAL` kept; missing `include_hwm` bootstraps from SEAL. Inclusion HWM: `scripthash.include_hwm`. **Leftover live OA** at `scripthash.head` (or non-`SHSR` `ovf/NNNNNN`): refuse — wipe `store/scripthash*` and restart with `--sh-index`. **SH head open:** sealed **main** shards load `.idx` only (one entry per 128 records; no fuse). Sealed **ovf** loads `.idx` + BF8R. Occupancy scan is not used on those files. A schema-14 page-era durable index is **refused** (wipe diff --git a/docs/env-knobs.md b/docs/env-knobs.md index d4309a857..9df3917df 100644 --- a/docs/env-knobs.md +++ b/docs/env-knobs.md @@ -68,7 +68,7 @@ for signet/mainnet sync. **Not** CLI. | `RBITCOIN_HEAD_SLOTS_TX` | Deleted; `tx.head` is segmented address head | | `RBITCOIN_SH_MAX_DIRECT_MERGE` | Deleted; catalog k-way materialize removed (unsorted shards are the tip path) | | `RBITCOIN_SH_MATERIALIZE` | Deleted; unsorted-shard collect/pack is the default (was env-select vs k-way) | -| `RBITCOIN_SH_RECOLLECT_WORKERS` | Deleted; unsorted collect is always nCPU | +| `RBITCOIN_SH_RECOLLECT_WORKERS` | Deleted; collect and pack share `RBITCOIN_SH_MERGE_WORKERS` (1.5 GiB per worker) | | `RBITCOIN_SH_RECOLLECT_SPILL_BYTES` | Deleted; catalog recollect/spill is gone (unsorted shards) | | `RBITCOIN_SH_TARGET_RUN_BYTES` | Deleted; catalog recollect/spill is gone | | `RBITCOIN_SH_MERGE_FANIN` | Deleted; no fan-in reduce | diff --git a/docs/external_findings/003-bip68-version-signedness-consensus-split.md b/docs/external_findings/003-bip68-version-signedness-consensus-split.md index 5e4186c6b..e2401ba0b 100644 --- a/docs/external_findings/003-bip68-version-signedness-consensus-split.md +++ b/docs/external_findings/003-bip68-version-signedness-consensus-split.md @@ -8,7 +8,7 @@ invalid. `oracle_consensus`) **Status:** fixed — BIP68 version gate uses unsigned compare (`bip68_active_for_tx`) -**Regression:** `rbitcoin-consensus` `block::tests::bip68_enforced_when_version_high_bit_set` — BIP68 active when version high bit set (`bip68_active_for_tx` unsigned compare). +**Regression:** `rbitcoin-consensus` `block::finality_tests::bip68_enforced_when_version_high_bit_set` — BIP68 active when version high bit set (`bip68_active_for_tx` unsigned compare). ## Summary diff --git a/docs/external_findings/009-witness-commitment-reserved.md b/docs/external_findings/009-witness-commitment-reserved.md index 6559d6e49..efd1ec677 100644 --- a/docs/external_findings/009-witness-commitment-reserved.md +++ b/docs/external_findings/009-witness-commitment-reserved.md @@ -5,7 +5,7 @@ **Severity:** medium — consensus accept-invalid **Status:** fixed — consensus reject on shipped path (2026-08-10) -**Regression:** `rbitcoin-consensus` `block::tests::s8_rejects_empty_or_multi_item_coinbase_witness_reserved`. +**Regression:** `rbitcoin-consensus` `block::structure_rule_tests::s8_rejects_empty_or_multi_item_coinbase_witness_reserved`. **Found by:** redteam-ecosystem harness (static analysis / code-review) ## Summary diff --git a/docs/external_findings/016-unknown-taproot-leaf-rejected.md b/docs/external_findings/016-unknown-taproot-leaf-rejected.md index b52483a13..d9bd3afad 100644 --- a/docs/external_findings/016-unknown-taproot-leaf-rejected.md +++ b/docs/external_findings/016-unknown-taproot-leaf-rejected.md @@ -3,7 +3,7 @@ **Component:** `rbitcoin-consensus` (`script/p2tr.rs::verify_script_path`) **Audit pin:** fuzzamoto report 005 / rbitcoin `8f3990f` **Severity:** critical — stalls on a block Core accepts (upgrade path) -**Status:** fixed — `script_path_accepts_unknown_taproot_leaf_version` +**Status:** fixed — `script_path_accepts_leaf_0x50_with_annex_and_0xc2` (leaf `0xc2` succeeds) **Found by:** fuzzamoto ## Summary diff --git a/docs/external_findings/048-standard-sigops.md b/docs/external_findings/048-standard-sigops.md index 973f36bd8..6c543c664 100644 --- a/docs/external_findings/048-standard-sigops.md +++ b/docs/external_findings/048-standard-sigops.md @@ -22,7 +22,6 @@ NULLFAIL, LOW_S, and CLEANSTACK stay at the flags production script verify already uses: NULLFAIL and LOW_S are off, and witness programs already require a clean true stack. Block validation does not gain those flags. -**Regression:** `rbitcoin-mempool` -`accept::tests::reject_tx_over_block_sigop_budget`, -`accept::tests::mempool_under_pressure`, +**Regression:** `rbitcoin-mempool` `mempool_accept_life` (`sigop_block_budget`), +`rbitcoin-net` `tx_relay` `mempool_under_pressure`, `rbitcoin-net` `peer::tests::invalid_script_is_scored_and_policy_is_not`. diff --git a/docs/external_findings/052-livera-review-index.md b/docs/external_findings/052-livera-review-index.md index 2b3937c9c..bc3c9851d 100644 --- a/docs/external_findings/052-livera-review-index.md +++ b/docs/external_findings/052-livera-review-index.md @@ -29,14 +29,14 @@ steps. | L4 | low | Empty median time panics | fixed | `empty_median_time_is_an_error` ([077](./077-empty-median.md)) | | L5 | low | Version nonce is not a CSPRNG | fixed | `rand_nonce_changes` ([078](./078-version-nonce.md)) | | L6 | low | Recent-reject set clears at the cap | fixed | `recent_reject_at_the_cap_does_not_clear` ([079](./079-recent-reject-cap.md)) | -| L8 | low | Manifest and txstat lengths allocate before a size check | fixed | `manifest_length_past_the_file_is_corrupt`, `txstat_blob_longer_than_the_file_is_corrupt` ([070](./070-manifest-length.md)) | +| L8 | low | Manifest and txstat lengths allocate before a size check | fixed | `map_prefix_longer_than_file_is_corrupt`, `txstat_blob_longer_than_the_file_is_corrupt` ([070](./070-manifest-length.md)) | | L9 | low | Tor control password on argv | fixed | `tor_control_password_rejects_a_line_break` ([080](./080-tor-control-password.md)) | | L10 | low | Datadir lock follows a symlink | fixed | `lock_file_does_not_follow_a_symlink` ([071](./071-datadir-lock-symlink.md)) | | L11 | low | Conf parse errors echo the raw line | fixed | `conf_error_names_the_file_and_line` ([081](./081-conf-error-line.md)) | | L12 | low | Invalid-hash set grows without a cap | fixed | `invalid_hash_set_stops_at_the_cap` ([082](./082-invalid-hash-cap.md)) | | L13 | low | Rate window grants two budgets at the boundary | fixed | `rate_limiter_boundary_does_not_grant_a_second_budget` ([059](./059-rate-window-boundary.md)) | | L14 | low | Mempool expiry runs only on admission | fixed | `expire_stale_drops_old_tx_without_a_new_accept`, `hub_live_journey` ([083](./083-mempool-expiry-cursor.md)) | -| L15 | low | Write jobs form a mutable slice over a shared buffer | fixed | pool write arm uses a shared slice ([072](./072-pool-write-slice.md)) | +| L15 | low | Write jobs form a mutable slice over a shared buffer | fixed | `io_session_pool` write arm (`from_raw_parts`) ([072](./072-pool-write-slice.md)) | | L2 | — | P2PKH fast path skips FindAndDelete | rejected | The fast path is the 25-byte template. A DER signature does not fit in that scriptCode, so FindAndDelete cannot change it. | | L3 | — | Witness-v0 strict DER independent of BIP66 | rejected | BIP141 witness verification is strict DER. Mainnet, testnet, signet, and regtest are unaffected. | | L7 | low | Rewind deeper than 1024 is refused | won't-fix | `REWIND_MAX_DEPTH` is a deliberate denial-of-service cap. Removing it needs an operator decision. No new knob. | diff --git a/docs/external_findings/070-manifest-length.md b/docs/external_findings/070-manifest-length.md index 464c3c8c0..7a8b43639 100644 --- a/docs/external_findings/070-manifest-length.md +++ b/docs/external_findings/070-manifest-length.md @@ -6,4 +6,4 @@ A sorted-run manifest and a txstat blob trusted a length and allocated it before noticing the file was shorter. A length past the remaining file or the fixed ceiling is corrupt and is not allocated. -**Regression:** `rbitcoin-store` `manifest_length_past_the_file_is_corrupt`, `txstat_blob_longer_than_the_file_is_corrupt`. +**Regression:** `rbitcoin-store` `map_prefix_longer_than_file_is_corrupt`, `txstat_blob_longer_than_the_file_is_corrupt`. diff --git a/docs/external_findings/072-pool-write-slice.md b/docs/external_findings/072-pool-write-slice.md index d1f4ad66f..8c7f5c6f0 100644 --- a/docs/external_findings/072-pool-write-slice.md +++ b/docs/external_findings/072-pool-write-slice.md @@ -6,4 +6,4 @@ The background IO worker formed a mutable slice for every job, including writes that only read the caller buffer. A write uses a shared slice. A read still uses a mutable slice. -**Regression:** `rbitcoin-store` pool write arm uses a shared slice. +**Regression:** no named unit test. The write arm in `rbitcoin-store` `io_session_pool` builds the caller buffer with `from_raw_parts` (shared); reads still use `from_raw_parts_mut`. diff --git a/docs/external_findings/README.md b/docs/external_findings/README.md index 6565b4c31..2d5586761 100644 --- a/docs/external_findings/README.md +++ b/docs/external_findings/README.md @@ -7,20 +7,20 @@ rbitcoin reference, or redteam static analysis). Numbered reports live beside th |----|----------|--------|--------|----------------------| | [001](./001-disconnect-on-invalid-block.md) | medium | Peer disconnect on invalid relayed block (BIP-152) | fixed | `rbitcoin-net` `peer::tests::cmpct_helpers_without_mempool_and_queue_out_closed` | | [002](./002-store-corrupt-record-on-invalid-block.md) | low | Invalid block misclassified as store corrupt | fixed | `rbitcoin-consensus` `error::tests::archive_unresolved_parent_is_missing_prevout_not_corrupt` | -| [003](./003-bip68-version-signedness-consensus-split.md) | high | BIP68 skipped for version with bit 31 set | fixed | `block::tests::bip68_enforced_when_version_high_bit_set` | +| [003](./003-bip68-version-signedness-consensus-split.md) | high | BIP68 skipped for version with bit 31 set | fixed | `block::finality_tests::bip68_enforced_when_version_high_bit_set` | | [004](./004-csv-nop-and-scriptnum-width.md) | high | CSV v1 no-op; CLTV/CSV 4-byte scriptnum | fixed | `script::interpreter::tests::csv_fails_when_tx_version_below_2` + Core script corpus | | [005](./005-non-topological-block-accepted.md) | high | Non-topological same-block spends accepted | fixed | `rbitcoin-test` `consensus_rules::header_and_spending_boundaries` | | [006](./006-p2sh-scriptsig-push-size.md) | medium | P2SH scriptSig pushes not limited to 520 bytes | fixed | `script::nested::tests::p2sh_scriptsig_push_over_520_rejected` | | [007](./007-p2sh-nested-witness-exactness.md) | medium | P2SH nested-witness scriptSig exactness / program rules | fixed | `script::nested` nested-witness malleation tests | | [008](./008-p2tr-keypath-sighash-zero.md) | medium | P2TR key-path 65-byte sig with sighash byte 0x00 | fixed | `script::p2tr::tests::key_path_rejects_65_byte_sighash_byte_zero` | -| [009](./009-witness-commitment-reserved.md) | medium | Witness commitment empty/multi-item coinbase witness | fixed | `block::tests::s8_rejects_empty_or_multi_item_coinbase_witness_reserved` | +| [009](./009-witness-commitment-reserved.md) | medium | Witness commitment empty/multi-item coinbase witness | fixed | `block::structure_rule_tests::s8_rejects_empty_or_multi_item_coinbase_witness_reserved` | | [010](./010-mempool-confirmed-spentness.md) | medium | Mempool no confirmed-chain spentness check | fixed | `rbitcoin-mempool` `accept::tests::reject_when_provider_has_no_unspent_coin` | | [011](./011-mempool-structural-chain-context.md) | medium | Mempool no structural chain-context validation | fixed | `accept::tests::reject_non_final_locktime_height`, `reject_immature_coinbase` | | [012](./012-p2sh-redeem-not-executed.md) | high | P2SH redeem skipped when BIP16 looks off | fixed | `bip16_from_prev_mtp_exception_and_time` | | [013](./013-bip68-unresolved-age-fail-open.md) | high | BIP68 unresolved coin age fails open | fixed | `bip68_unresolved_coin_age_fails_closed` | | [014](./014-stranded-on-peer-reorg.md) | high | Stranded when peer reorgs (sync) | fixed | `peer_catchup_compact_reorg` | | [015](./015-spend-rejected-block-outputs.md) | high | Spend outputs of a rejected block | fixed | cluster 017/019 + structural fail-closed | -| [016](./016-unknown-taproot-leaf-rejected.md) | critical | Unknown tapleaf version rejected | fixed | `script_path_accepts_unknown_taproot_leaf_version` | +| [016](./016-unknown-taproot-leaf-rejected.md) | critical | Unknown tapleaf version rejected | fixed | `script_path_accepts_leaf_0x50_with_annex_and_0xc2` | | [017](./017-duplicate-txid-unconnected-instance.md) | medium | Txid resolve hits unconnected instance | fixed | `resolve_txid_prefers_connected_over_newer_unconnected` | | [018](./018-compact-block-duplicate-tx.md) | high | Compact block duplicates a tx | fixed | `repeated_short_id_is_requested_not_duplicated` | | [019](./019-bip30-not-enforced.md) | critical | BIP30 not enforced | fixed | cluster 015/017 + BIP34-gated batch | @@ -47,7 +47,7 @@ rbitcoin reference, or redteam static analysis). Numbered reports live beside th | [037](./037-rpc-esplora-limits.md) | high | RPC body, work queue, waits, and Esplora client id | fixed | `unauthorized_large_content_length_is_401_before_body` | | [039](./039-io-lifetimes.md) | high | Store I/O buffer must outlive the submit | fixed | `enter_failure_with_pending_matches_the_hard_cap` | | [047](./047-orphan-reserve.md) | medium | Orphan reserve can refuse every later orphan | fixed | `mempool_under_pressure` | -| [048](./048-standard-sigops.md) | medium | No standard sigop cap before the interpreter | fixed (block limit, not 16k) | `reject_tx_over_block_sigop_budget`, `mempool_under_pressure` | +| [048](./048-standard-sigops.md) | medium | No standard sigop cap before the interpreter | fixed (block limit, not 16k) | `mempool_accept_life` (`sigop_block_budget`), `mempool_under_pressure` | | [049](./049-rolling-min-fee.md) | medium | Full-mempool fee floor is a static bump | fixed | `mempool_under_pressure` | | [050](./050-cluster-once.md) | low | Cluster rebuild once per input | fixed | `mempool_under_pressure` | | [051](./051-rbfr-direct-set.md) | low | RBFR uses the direct conflict set | won't-fix | `mempool_accept_life` | @@ -74,9 +74,9 @@ rbitcoin reference, or redteam static analysis). Numbered reports live beside th | [067](./067-fuse8-segment.md) | medium | fuse8 segment length must be a power of two | fixed | `fuse8_segment_length_must_be_power_of_two` | | [068](./068-published-end.md) | medium | Body reads compare the caller's published end | fixed | `body_read_past_published_end_is_corrupt` | | [069](./069-uring-drop-drain.md) | high | io_uring drop does not free a buffer still in the kernel | fixed | `drain_guard_drop_with_leftover_pending_does_not_abort` | -| [070](./070-manifest-length.md) | low | Manifest and txstat lengths must fit the file | fixed | `manifest_length_past_the_file_is_corrupt` | +| [070](./070-manifest-length.md) | low | Manifest and txstat lengths must fit the file | fixed | `map_prefix_longer_than_file_is_corrupt`, `txstat_blob_longer_than_the_file_is_corrupt` | | [071](./071-datadir-lock-symlink.md) | low | Datadir lock does not follow a symlink | fixed | `lock_file_does_not_follow_a_symlink` | -| [072](./072-pool-write-slice.md) | low | Pool write jobs use a shared slice | fixed | `pool write arm uses a shared slice` | +| [072](./072-pool-write-slice.md) | low | Pool write jobs use a shared slice | fixed | `io_session_pool` write arm (`from_raw_parts`) | | [073](./073-testnet-milestone-scripts.md) | medium | Omitted testnet milestone checks every script | fixed | `p3_default_milestone_heights` | | [074](./074-tip-body-orphan-bytes.md) | medium | Tip-follow bodies and one peer's orphans are bounded | fixed | `pending_block_over_four_megabytes_is_not_parked` | | [075](./075-secret-debug.md) | low | Secret debug output is redacted | fixed | `debug_does_not_print_secret_bytes` | diff --git a/docs/invariants.md b/docs/invariants.md index da414478d..0dd1da7c9 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -17,7 +17,7 @@ with `Err(…Corrupt("invariant: …"))` (and `debug_assert!` where useful). Do | **Load miss** | Spend annotate without `spent_range` abs; body decode without idx range; pin without outs for need_vouts; ensure without abs for a spend edge | Assert / hard Err; fix lookup/load | | **Environment** | bulk IO backend uring vs pread/pwrite (single backend trait) | Keep modality only | | **Protocol** | BIP30 multi-spender confirmed-strong walk; same-block spends; coinbase null create | Real branches (not soft recovery) | -| **Format migrate** | Occupied 22 rewrites `create.loc.ovf` 12 B→16 B; occupied 23 rewrites `header.body` 88 B→96 B; occupied 24 zero-extends `txstat.body`; `meta` to live `SCHEMA_VERSION`. Leftover fuse8 v1 / flat idx / Shared SH body / Paged pack8 **refuse** | Soft migrate (tmp+rename) or explicit wipe/rebuild | +| **Format migrate** | Occupied 22 rewrites `create.loc.ovf` 12 B→16 B; occupied 24/25 strips `header.body` 96 B→88 B (an 88 B body stays 88 B); occupied 24 zero-extends `txstat.body`; `meta` to live `SCHEMA_VERSION`. Leftover fuse8 v1 / flat idx / Shared SH body / Paged pack8 **refuse** | Soft migrate (tmp+rename) or explicit wipe/rebuild | | **API / product** | RPC body from store; Electrum mempool after chain; compact → getdata | Keep | **One path.** Do not reintroduce a silent colder alternate when lookup or load diff --git a/docs/io-modality.md b/docs/io-modality.md index 0db43c8ea..7b37ce61f 100644 --- a/docs/io-modality.md +++ b/docs/io-modality.md @@ -41,7 +41,7 @@ Unknown tokens (including deleted `mmap`) fall through to the platform default. | `iocp` | Windows IOCP (Windows **default**) | | `pread` / `fd` / `libc` / `pwrite` | Disable session; libc positional IO (+ workers for reads) | -**Defaults:** Linux `io_uring` if the ring opens, else pool. Darwin **pool**. +**Defaults:** Linux `io_uring` if the ring opens, else libc **pread** (session off). Darwin **pool**. Windows **IOCP**. Windows IoRing is not supported. **Windows table handles** are `FILE_FLAG_OVERLAPPED` so IOCP can bind. @@ -149,7 +149,7 @@ RAM fence (~15 MiB at 1M blocks), not a file. | Block index build (filters + tweaks) | `RBITCOIN_IO` | `read_index_window`: one held TLS session per window, four stages (`create.loc` / `input.loc` / `seqsigwit.loc` windows → block `txout` / `input.body` / `txid.body` spans → parent `create.loc` + P2TR-output `seqsigwit.body` + parent `txid.body` → parent `txout`), offsets from RAM checkpoints so nothing nests. Parents read once per window. Short/errno on a live session libc-completes; no session → serial pread through the same stages. Writes are not on the ring: one filter put and one tweak put per window. | | SP-tweak hole serve | `RBITCOIN_IO` | `tweaks_for_height` for heights the index has not sealed: idx ranges **before** TLS ring; uring/pread `txout.body`, then `seqsigwit` + parent `txout` for P2TR only. | -Default: Linux uring if the ring opens else pool; Darwin pool; Windows +Default: Linux uring if the ring opens else libc pread; Darwin pool; Windows IOCP. Ring depth **128** (merge may grow). `RBITCOIN_IO=pread` forces libc. ### Table transport (all fd) diff --git a/docs/operator/field-notes.md b/docs/operator/field-notes.md index babade0c8..1dd6d4a52 100644 --- a/docs/operator/field-notes.md +++ b/docs/operator/field-notes.md @@ -76,9 +76,10 @@ non-empty `header.body` is refused (wipe those files and reindex). `--max-outbound` / `max_outbound` is the IBD download peer count (default **16**). IBD `target_peers` is that value clamped to **8..=32**, so `--max-outbound 4` -still dials 8 catch-up peers. Concurrent block getdata is about `N × 16` -(`IbdConfig::per_peer` is code-only 16, Core-like; there is no -`--maxblocksperpeer`). +still dials 8 catch-up peers. Concurrent block getdata is about `N × 64` +(`IbdConfig::per_peer` is code-only 64; a 16 MiB payload cap stops one +peer holding 64 large bodies; the 1024-hash window still bounds the total; +there is no `--maxblocksperpeer`). On a typical home uplink use **8**. That is also the floor on a tight link: more peers will not raise a saturated wire and can make `relative-slow` peel a diff --git a/docs/operator/interfaces.md b/docs/operator/interfaces.md index 9483261c4..599762447 100644 --- a/docs/operator/interfaces.md +++ b/docs/operator/interfaces.md @@ -95,7 +95,7 @@ Re-measure fat keys on the operator host (`rbitcoin-bench --suite casa ### App DoS floor (always on) -Shared [`ServeLimits`](crates/rbitcoin-electrum) defaults (also the future Esplora +Shared [`ServeLimits`](../../crates/rbitcoin-electrum) defaults (also the future Esplora floor). Excess connections are **rejected immediately** (no hang); oversize lines and idle clients fail closed. @@ -202,7 +202,7 @@ Leave `--max-sh-creates` at **10000** (or **0** for an unlimited unpaged join) f TCP Esplora does not keep a mempool of `Arc`. `GET /mempool` loads the fee snapshot (count/vsize/total_fee + histogram). Unix `/internal` mempool-tx pages lazy-build one published body snapshot (JSON `OnceLock` per live tx after -the first page; dirty/singleflight; not FIFO/LRU). RAM: [`docs/ibd-memory.md`](docs/ibd-memory.md). +the first page; dirty/singleflight; not FIFO/LRU). RAM: [`docs/ibd-memory.md`](../ibd-memory.md). | Feature | Behavior | |---------|----------| diff --git a/docs/operator/operations.md b/docs/operator/operations.md index 6dfff3429..4b99ec6c9 100644 --- a/docs/operator/operations.md +++ b/docs/operator/operations.md @@ -51,7 +51,7 @@ RPC auth is a unix socket (`--rpc`) or Bearer `{datadir}/rpc.token` (TCP). There is no `--rpcuser` / `--rpcpassword`. Core names (`-maxconnections`, `-whitelist`, `-blocksonly`, `-minimumchainwork`, …) are translated by the functional `bitcoind` shim only -([`docs/core-functional.md`](docs/core-functional.md)). +([`docs/core-functional.md`](../core-functional.md)). Clean smoke: @@ -84,7 +84,7 @@ Clean smoke: | `--milestone HEIGHT` | `milestone=` | mainnet anchor at 840000; testnet3 anchor at 2500000; signet 0; explicit height is height-only | | `--max-outbound N` | `max_outbound=` | 16 live download peers | | `--max-inbound N` | `max_inbound=` | 125 inbound sessions; **0** = no inbound slots (outbound-only) | -| `--mempool-size-mb N` | `mempool_size_mb=` | ~300 MiB weight | +| `--mempool-size-mb N` | `mempool_size_mb=` | **300** → 300e6 WU (`N × 1_000_000` weight, not MiB) | | `--conf FILE` | | none | | `--log-level LEVEL` | `log_level=` | `info` | | `--api-log PATH` | `api_log=` | off — JSONL of Electrum / Esplora / RPC calls | @@ -292,14 +292,14 @@ tip: best= height= version= tx= date= Emitted from the tip-follow / wire accept path (`ChainHub::connect_at`). IBD bulk confirm does **not** spam this line per block — use the periodic IBD status below. Core functional tests still grep `UpdateTip: …` via the debug.log map -([`docs/core-functional.md`](docs/core-functional.md)). +([`docs/core-functional.md`](../core-functional.md)). ### Tip-follow status lines (after catch-up + tip SH ready) | Line | Level | Use | |------|-------|-----| | `tip: perf` | DEBUG | One JSON object every ~5s: follow peers, blocks this window, mempool accept/reject + wall µs, inv/getdata/announce, Esplora/Electrum req counts + avg/max µs, historical block serve (`serve_n`, `serve_bytes`, `serve_tx`, `serve_avg_us`, `serve_max_us`), and SV2 (`sv2_checks`, `sv2_builds`, `sv2_build_avg_us`, `sv2_build_max_us`; zeros without `--sv2-tp-listen`) | -| `tip: accept` | INFO | Per accepted tip block: wall/load/script/class_a/class_c/SH plus lookup/struct/drain/mp_strip/other (not emitted on reject) | +| `tip: accept` | DEBUG | Per accepted tip block: wall/load/script/class_a/class_c/SH plus lookup/struct/drain/mp_strip/other (not emitted on reject) | | `tip: best=` | INFO | New best hash/height after connect | | `cmpct reconstruct` | INFO | Per compact reconstruct: fill sources (`prefill`/`mempool`/`extra`/`orphan`) and `fetched=` `blocktxn` count/bytes. `fetched=0/0` means no getblocktxn round-trip. Getdata fallback: `getdata missing=` | | `node: tip=…` | DEBUG | Same height change plus `follow_live` (use `tip: best=` at info) | @@ -319,10 +319,10 @@ Requires **tip mode** (`node: catch-up complete … tip tracking`). During IBD u | Line | Level | Use | |------|-------|-----| -| `ibd: progress` | INFO | Tip rate, `loadq`/`scriptq`/`writeq`, `txs=` (Class A / `tx.idx` count), horizon, tip ETA, **`bq soft=n/win RAM=`** (in-RAM body queue; soft densify: under ~100 MiB free ahead, over that only ~1 min confirm window, at/over 1 GiB assign-stop holes within that window and not past fetched_hi) | +| `ibd: progress` | INFO | Tip rate, `loadq`/`scriptq`/`writeq`, `txs=` (Class A create count), horizon, tip ETA, **`bq soft=n/win RAM=`** (in-RAM body queue; soft densify: under ~100 MiB free ahead, over that only ~1 min confirm window, at/over 1 GiB assign-stop holes within that window and not past fetched_hi) | | `ibd: perf` | DEBUG | One JSON object of the 5s sample (inflight, queues, stage times, RSS, work path, confirm pipe). Zeros are included. `load_ms` is pin+assemble only | -Default INFO is `ibd: progress` only. `--log-level debug` adds the JSON sample. `tip: accept` is the same kind of JSON line at DEBUG; `tip: best` stays INFO. Pipeline roles: [`docs/concurrency.md`](docs/concurrency.md). Head files: [`docs/heads.md`](docs/heads.md). +Default INFO is `ibd: progress` only. `--log-level debug` adds the JSON sample. `tip: accept` is the same kind of JSON line at DEBUG; `tip: best` stays INFO. Pipeline roles: [`docs/concurrency.md`](../concurrency.md). Head files: [`docs/heads.md`](../heads.md). `pin_txid%` is stamp `txid→create_fk` from the load-batch skeleton vs leftover `tx.head` (IBD skeleton path should stay at 100%). `pin_hit%` is load outs adopt/plan reuse — this-window range-fills are `pin_new` only. @@ -350,9 +350,9 @@ from the race who still owe it. When `hole=` is 0, at most one extra racer is added on the first later gap in the 32-window, and only if that owner is missing, has held the hash ≥30s, or ≤ pack-median/4. -Densify default is 8 in-flight hashes per peer (none while a quarter-full tip -hole is open, so getdata queues can drain for tip+1); -16 only for an EWMA outlier at ≥ 2× pack median. WARN +Densify default is 32 in-flight hashes per peer (half of the 64-block +per-peer ceiling; none while a quarter-full tip hole is open, so getdata +queues can drain for tip+1); 64 only for an EWMA outlier at ≥ 2× pack median. WARN `ibd: peer[…] stalled` is 30s without qualifying rx (≥64 KiB stream or a block / decode-fail / NotFound event) after work start. WARN `ibd: peer[…] relative-slow` is a quarter-median outlier (cluster gate keeps a @@ -370,7 +370,7 @@ uniformly slow pack). Slow or constrained uplinks: [Slow / constrained uplink (deepest-cand-first). **Class A `txout` / `seqsigwit` / `spent` + `create.loc` / `seqsigwit.loc`, `tx.head`, header head, SH head/body, and spenders are fd pread/pwrite**. -Full modality matrix: [`docs/io-modality.md`](docs/io-modality.md). +Full modality matrix: [`docs/io-modality.md`](../io-modality.md). ## Health probes and metrics @@ -547,9 +547,9 @@ scrape_configs: | Script templates | allow if consensus-valid (within weight/CPU) | | RBF | **full RBF** (no BIP125 signaling required) | | Annex | empty OK; non-empty only if first data byte after `0x50` is `0x00` | -| Cluster caps | 64 txs / 101 kWU | +| Cluster caps | 64 txs / 101 kvB (404 kWU) | | Eviction | worst linearization **chunk** when over weight budget | -| Fee estimate | **10-minute inclusion** (cluster-chunk frontier + confirm-memory floor); see [`docs/mempool-fee-estimation.md`](docs/mempool-fee-estimation.md) | +| Fee estimate | **10-minute inclusion** (cluster-chunk frontier + confirm-memory floor); see [`docs/mempool-fee-estimation.md`](../mempool-fee-estimation.md) | | Compaction | DEAD slots reclaimed when wasteful (auto after confirm removes) | | Slot table | **131 072** initial records (grows by doubling to 1 048 576); free-slot ensure **before** append | @@ -575,8 +575,8 @@ not a full reindex. Widespread mid-chain header graph poison still means a clean datadir. **Mempool recovery:** `{datadir}/mempool/` is a private sidecar (not Class A), -schema **2**. Leftover schema **1** (pre-packed `fee‖weight‖bitcoin-serialize`) -converts to packed on the next open (Class A untouched). Wipe `{datadir}/mempool/` +schema **3**. Leftover schema **1** (pre-packed `fee‖weight‖bitcoin-serialize`) +or **2** converts to schema 3 on the next open (Class A untouched). Wipe `{datadir}/mempool/` only if the sidecar is damaged or an unknown schema/old 4k-slot table was left wedged — the next start recreates it empty and redownloads unconfirmed txs. Do **not** wipe `store/` for mempool slot/full/schema errors. @@ -589,7 +589,7 @@ Do **not** wipe `store/` for mempool slot/full/schema errors. is on DEBUG `tip: perf` (`serve_n`, `serve_bytes`, `serve_tx`, `serve_avg_us`, `serve_max_us`), not a per-block line. Host throughput probe: `python3 scripts/ibd-serve-bench.py 127.0.0.1:8333` (needs `cryptography` - and a BIP324 client; see [`TESTING.md`](TESTING.md) § P2P serve bench). + and a BIP324 client; see [`TESTING.md`](../../TESTING.md) § P2P serve bench). - **Discovery** queries Core DNS seeds for `NETWORK|WITNESS|P2P_V2` (`x809.` first; the bare seed name only if that returns nothing). Learned `addr` / `addrv2` is ingested only when the row advertises `P2P_V2` diff --git a/docs/operator/setup.md b/docs/operator/setup.md index 60db0e61c..5fadbb6fd 100644 --- a/docs/operator/setup.md +++ b/docs/operator/setup.md @@ -14,8 +14,8 @@ full scripts. Signet’s default is **0** (every script, slower on purpose). Architecture and confirm pipeline: [`docs/architecture.md`](../../docs/architecture.md), [`docs/concurrency.md`](../../docs/concurrency.md). Download defaults to **1024** -concurrent getdata (not a tip-distance cap), max **16** blocks in transit per -peer. +concurrent getdata (not a tip-distance cap), max **64** blocks in transit per +peer (a 16 MiB payload cap stops one peer holding 64 large bodies). ## Build @@ -161,7 +161,7 @@ store IO, mmap sealed fuse, a few-block query confirm, `--smoke`); they do not upload binaries. Local Linux `target/release/` install is still `nix build .#rbitcoin-musl` on a clean master tree. Windows IoRing is not supported. Darwin/Windows are not Nix packages — see -[`docs/reproducible-builds.md`](docs/reproducible-builds.md). +[`docs/reproducible-builds.md`](../reproducible-builds.md). **Darwin Gatekeeper:** the Darwin binaries are ad-hoc signed (`codesign -s -`), not notarized. If Finder or a browser sets quarantine and the binary is killed diff --git a/docs/operator/storage.md b/docs/operator/storage.md index 64f87d095..31f117953 100644 --- a/docs/operator/storage.md +++ b/docs/operator/storage.md @@ -2,9 +2,9 @@ ## Bulk store IO backends -**Bulk batch** uses a **single** switch: `RBITCOIN_IO=uring|pread` (default uring +**Bulk batch** uses a **single** switch: `RBITCOIN_IO=uring|pool|iocp|pread` (default uring when available). Table transport is always **fd pread/pwrite**. Compact Class C -is L2 write-behind; see [`docs/io-modality.md`](docs/io-modality.md). Per-path +is L2 write-behind; see [`docs/io-modality.md`](../io-modality.md). Per-path env overrides are **removed**. If `uring` is selected but setup fails, demote to **pread** / **pwrite**. If a live ring stops completing (`drain slow`, then abort), restart with **`RBITCOIN_IO=pread`** — the process does not switch backends itself. @@ -13,24 +13,24 @@ restart with **`RBITCOIN_IO=pread`** — the process does not switch backends it |-----|--------|------| | **`RBITCOIN_IO`** | `uring` \| `pool` \| `iocp` \| `pread` | Only bulk switch | -Inventory / survivors: [`docs/env-knobs.md`](docs/env-knobs.md). -Token meanings and ring depth: [`docs/io-modality.md`](docs/io-modality.md). -`RWF_DONTCACHE` is not used ([`SCHEMA.md`](SCHEMA.md) Schema 17 freeze). +Inventory / survivors: [`docs/env-knobs.md`](../env-knobs.md). +Token meanings and ring depth: [`docs/io-modality.md`](../io-modality.md). +`RWF_DONTCACHE` is not used ([`SCHEMA.md`](../../SCHEMA.md) Schema 17 freeze). ## Defaults and memory budgets | Knob | Default | Override | |------|---------|----------| | IBD concurrent getdata | **1024** | code `IbdConfig::window` | -| Blocks in transit / peer | **16** | `IbdConfig::per_peer` | +| Blocks in transit / peer | **64** | `IbdConfig::per_peer` (16 MiB payload cap per peer) | | Live IBD peers | **16** | `--max-outbound` | | Inbound P2P sessions | **125** | `--max-inbound`. At capacity, unprotected inbounds are evicted. Incomplete VERSION/VERACK is dropped after **60 s** (releases the slot). | | Milestone (script skip) | mainnet anchor **840000**, testnet3 anchor **2500000**, signet **0**, regtest 0 | `--milestone` (`0` = full scripts; explicit height is height-only) | | ConfirmParentCache header plans | always on | Tip-ahead header + tx_fks for multi-block MTP (no create pin FIFO) | -| Bulk store IO | **uring** (Linux) when available | `RBITCOIN_IO` only. Matrix: [`docs/io-modality.md`](docs/io-modality.md) | -| Archive Class A append | **pwrite** (always) | `txout` / `seqsigwit` / `spent` + `*.idx` | -| `tx.head` (segmented) | fixed geometry | Default **25-bit**. Rebuild env: [`docs/env-knobs.md`](docs/env-knobs.md). Bytes: [`SCHEMA.md`](SCHEMA.md) / [`docs/heads.md`](docs/heads.md). Legacy mono-head datadirs require reindex | -| Confirm stages | **lookup · load · scripts · write** | Queues and pack/wave: [`docs/concurrency.md`](docs/concurrency.md). RAM: [`docs/ibd-memory.md`](docs/ibd-memory.md) | +| Bulk store IO | **uring** (Linux) when available | `RBITCOIN_IO` only. Matrix: [`docs/io-modality.md`](../io-modality.md) | +| Archive Class A append | **pwrite** (always) | `txout` / `seqsigwit` / `spent` plus `create.loc` / `seqsigwit.loc` | +| `tx.head` (segmented) | fixed geometry | Default **25-bit**. Rebuild env: [`docs/env-knobs.md`](../env-knobs.md). Bytes: [`SCHEMA.md`](../../SCHEMA.md) / [`docs/heads.md`](../heads.md). Legacy mono-head datadirs require reindex | +| Confirm stages | **lookup · load · scripts · write** | Queues and pack/wave: [`docs/concurrency.md`](../concurrency.md). RAM: [`docs/ibd-memory.md`](../ibd-memory.md) | | Confirm batch inputs | **8000** soft | Hardcoded. Live line: `h= n= in=` (**n** = blocks in pack, **in** = Σ inputs) | | Mempool weight budget | **~300e6 WU** | `--mempool-size-mb N` (maps N×1e6 WU) | | Inhibit auto-suspend | **off** | `--inhibit-suspend` (uses `systemd-inhibit` if available) | @@ -50,7 +50,7 @@ without clearing known flags. New writes are `rbitcoin-peers-v2` (IPv4, IPv6, and Tor v3 `.onion:port` tokens). `rbitcoin-peers-v1` IPv4/IPv6 files still load. -**Index modes:** Direct vs Tip: [`docs/concurrency.md`](docs/concurrency.md). +**Index modes:** Direct vs Tip: [`docs/concurrency.md`](../concurrency.md). IBD finishes Class A + `tx.head` + spend annotations **before** tip; tip entry does not backfill them. Scripthash: durable head stays Tip write-behind; no head means Direct defers SH until a Class A collect + unsorted pack at @@ -59,7 +59,7 @@ horizon. Confirm does **not** enqueue SH during Direct. SIGINT keeps every sealed SH head; resume packs only unsealed shards (holes stay; missing `DONE` restarts collect). `RBITCOIN_SH_FORCE_REBUILD=1` wipes the head and does a full Class A collect + unsorted pack — unset after success -([`docs/env-knobs.md`](docs/env-knobs.md)). Missing `include_hwm` bootstraps +([`docs/env-knobs.md`](../env-knobs.md)). Missing `include_hwm` bootstraps from SEAL (never clamp SEAL→0). Clearing residual run files **preserves `SEAL`**. **SIGINT** mid cold keeps finished prefix shards (`scripthash.cold_progress`). Materialize status logs ~**every 10s**. @@ -68,7 +68,7 @@ On enter Direct, leftover `ibd_utxo.map` / `point.runs` / `tx.runs` from old Catchup datadirs are removed — prefer a **fresh datadir**. Layout, shard counts, ingest OA, and refuse lines: [`SCHEMA.md`](../../SCHEMA.md) and [Schema upgrade](#schema-upgrade). Working-set sizes: SCHEMA census and -[`docs/ibd-memory.md`](docs/ibd-memory.md). +[`docs/ibd-memory.md`](../ibd-memory.md). ## Schema upgrade @@ -127,7 +127,7 @@ Class C. Restart the same binary: `tx.head` rebuilds from Class A; with `--sh-index`, SH rematerializes. Do **not** `rm -rf store/`. **Kill-9 / crash is not a schema upgrade.** Open follows -[`docs/crash-recovery.md`](docs/crash-recovery.md) (tip-as-commit, Class C +[`docs/crash-recovery.md`](../crash-recovery.md) (tip-as-commit, Class C repair above tip). Prefer SIGTERM ([Resume / clean stop](#resume--clean-stop)). A corrupt file still means wipe/reindex — not an in-process repair. @@ -140,7 +140,7 @@ Core `-txindex` (we always keep Class A + `tx.head` for by-txid lookup). | Mode | Behavior | |------|----------| | **off (default)** | No SH run enqueue during IBD; no tip bulk materialize. Tip follow + mempool relay + JSON-RPC work without SH. | -| **on** (`--sh-index` / `sh_index=1`) | Direct IBD SH runs + tip bulk materialize; address/scripthash Electrum/Esplora methods work. | +| **on** (`--sh-index` / `sh_index=1`) | Direct IBD defers SH. Tip entry runs two Class A scans, then pack and seal; write-behind follows. Address/scripthash Electrum/Esplora methods work after that seal. | Electrum/Esplora **start without** `--sh-index`. Address/scripthash methods then fail closed (`scripthash index disabled`). Txid/outpoint/block/fees still work. Matrix: [`docs/lightning.md`](../../docs/lightning.md). @@ -168,7 +168,7 @@ Tip-follow readiness is **independent** of SH materialize (`tip_follow_ready` Keep **`store/scripthash.unsorted/`** until every shard has `scripthash.head/NN.packed`. That mark is the pack commit. Pass-1 `.mphf`+`.val` without it is not sealed. Resume rules: -[`docs/crash-recovery.md`](docs/crash-recovery.md) (scripthash cold resume). +[`docs/crash-recovery.md`](../crash-recovery.md) (scripthash cold resume). Extra disk during build is **`SHKSP01` spills** plus **`SHPST01` post spills**. Restart with the same `--datadir --sh-index`. Do not delete unsorted files to start over unless you intend a full Class A collect @@ -182,12 +182,12 @@ to start over unless you intend a full Class A collect | `DONE.post`, some `.packed` | Pack only the unmarked shards. | | All `.packed` | Tip write-behind. A second start does not collect. Electrum stays down until `include_hwm` covers the tip, then this process binds it. | | Complete head, no marks, no extract | Soft-migrate: write `.packed`. Missing `include_hwm` is set from the create count. | -| Kill-9 mid pack | Unfinished shard is redone. Open follows [`docs/crash-recovery.md`](docs/crash-recovery.md). | +| Kill-9 mid pack | Unfinished shard is redone. Open follows [`docs/crash-recovery.md`](../crash-recovery.md). | | Corrupt SH (leftover live OA, mixed body, refuse line) | Wipe `store/scripthash*` only, keep Class A, rematerialize with `--sh-index`. | Electrum waits until SH is tip-ready. Do **not** `rm -rf store/` for an SH abort. Force-rebuild sticky env (`RBITCOIN_SH_FORCE_REBUILD`) must never redo -multi-hour Class A work casually — [`docs/env-knobs.md`](docs/env-knobs.md). +multi-hour Class A work casually — [`docs/env-knobs.md`](../env-knobs.md). ## Silent payment tweaks (`--sp-tweaks`) diff --git a/docs/peer-clients.md b/docs/peer-clients.md index 7329c6125..79c917e0d 100644 --- a/docs/peer-clients.md +++ b/docs/peer-clients.md @@ -159,7 +159,7 @@ than a TUI. | Idea | Why it fits | Why not blindly | |------|-------------|-----------------| -| **Self-authenticating BIP352 tweak rows** (block hash in the row; stream/replay without trusting the server) | We already have `--sptweaks` and Electrum `tweaks.subscribe` ([`OPERATOR.md`](../OPERATOR.md)). Their row shape is a better serve contract than “trust the node.” | Do not make SH/tweaks wait on Class A if that fights write-behind. | +| **Self-authenticating BIP352 tweak rows** (block hash in the row; stream/replay without trusting the server) | We already have `--sp-tweaks` and Electrum `tweaks.subscribe` ([`OPERATOR.md`](../OPERATOR.md)). Their row shape is a better serve contract than “trust the node.” | Do not make SH/tweaks wait on Class A if that fights write-behind. | | **`/healthz` + `/readyz` + Prometheus** | Operators and k8s probe these. Logs are not a probe. | Unauthenticated loopback; do not grow a second metrics dialect. `GET /progress` is not one: it is the always-on human view of the long-stage counters that `/metrics` (opt-in) also exports ([`operations.md`](./operator/operations.md#progress)). | | **Explicit reorg log** (`reorg.log` + `getreorghistory`) | Tip-follow already disconnects hopeless forks ([`ibd-memory.md`](./ibd-memory.md)); a durable ring is cheap evidence. | Webhooks / MCP / TUI are satd product, not ours. | | **SIGHUP for hot mempool/relay knobs** | Useful once we have more policy flags. | We already log to stdout; do not cargo-cult Core `debug.log` reopen. satd’s split (`SIGHUP` config, `SIGUSR1` TLS) is the right shape **if** we add native TLS. | diff --git a/docs/quality.md b/docs/quality.md index 2e10038f6..624d3c77d 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -37,8 +37,7 @@ Counts and procedures stay in the owner doc. A row is the rank and the outcome. | 3 | **Q-31** | Hermetic tip fixtures | Frozen signet/mainnet tip packs for offline consensus/Electrum regression (no live API). Fuzz already merges tiny `signet_block_*.bin` / `mainnet_block_290329.bin`. Electrum hermetic packs still Open. | | 4 | **R-10** | Residual god-files | Peel **only** when a higher row needs a seam. Do not split `interpreter.rs` opcode `match` or io_uring machines. Named extracts: **Q-61** Completed. **Q-56** Completed (scriptnum + pack-ints). **Q-68** Completed (create.loc SIMD). | | 5 | **Q-67** | `asked_blocks` clone on hold | `hold_body` clones `asked_blocks` before `held_bodies` insert so the read lock does not overlap the write (`HeldBodies::insert` already takes `&HashSet`). Bound is `MAX_SERVE_BLOCKS` × peers. Follow-up: pass the read guard with a documented lock order, or keep the clone as a named trade. Owner: `crates/rbitcoin-net/src/chain.rs`. | -| 6 | **Q-69** | CLN / LDK chain backend | Operator can point CLN `bcli` and ldk-node Esplora/Electrum at this node. Owner: [`lightning.md`](./lightning.md). | -| 7 | **Q-70** | Batch lookup-path hit counters if a profile names them | `head_resolve_stats::add_hit_rank` does two relaxed `fetch_add`s per resolved txid (`head_resolve_denserels`, `tx_table`). SH extract collect flushes output and hit counts once per fk batch, not per output. Still once per event: `accepted_wb` (`SeqCst`, per confirmed block), `serve_perf::note_serve` (per historical getdata), `page_ios` (per SH page read). Done: a profile shows `add_hit_rank` on confirm lookup and those counters flush per batch, or the profile shows the line is noise and this row moves to Won't-fix. `add_hit_ages` is already a batched flush; `add_hit_age` remains the one-bucket path. | +| 6 | **Q-70** | Batch lookup-path hit counters if a profile names them | `head_resolve_stats::add_hit_rank` does two relaxed `fetch_add`s per resolved txid (`head_resolve_denserels`, `tx_table`). SH extract collect flushes output and hit counts once per fk batch, not per output. Still once per event: `accepted_wb` (`SeqCst`, per confirmed block), `serve_perf::note_serve` (per historical getdata), `page_ios` (per SH page read). Done: a profile shows `add_hit_rank` on confirm lookup and those counters flush per batch, or the profile shows the line is noise and this row moves to Won't-fix. `add_hit_ages` is already a batched flush; `add_hit_age` remains the one-bucket path. | R-ids were the 2026-08-12 slice. Canonical id is **bold**. Do not start **R-11+**. Next unused Q-id is **Q-73**. diff --git a/docs/road-to-1.0.md b/docs/road-to-1.0.md index ab598c5e7..67c4bb217 100644 --- a/docs/road-to-1.0.md +++ b/docs/road-to-1.0.md @@ -27,7 +27,7 @@ Core clone, not a soak badge, not a desktop wallet. | **Datadir** | Every **1.x.x** opens a **1.0.0** store. No silent wipe. | | **Chain** | No known consensus divergences from Core. | | **Wallets** | Electrum / Esplora work for the clients we claim, on a node that finished IBD + SH index. | -| **Index time** | `--shindex` after IBD is a known, resume-safe wait — not an unbounded hour-loss. Faster than 0.5 is the point. | +| **Index time** | `--sh-index` after IBD is a known, resume-safe wait — not an unbounded hour-loss. Faster than 0.5 is the point. | | **IBD** | Typical block connect about **1 s** on a laptop-class SSD. | | **RAM** | **2 GiB** process RSS is enough for IBD, SH build, and tip-follow (knobs may trade wall time, e.g. serial SH build). Heap, not “the disk is in page cache.” | | **Fees** | The 10-minute inclusion estimate is aimed at txs that actually get in, not Core’s historical estimator. | @@ -61,7 +61,7 @@ taking on for 1.0. `--prune-seqsigwit` is in (seqsigwit-window prune, COMPAT **done** RPCs, P2P, and mempool should be covered by the Core functional harness **or** an explicit “we differ on purpose” note (fee -product, error codes, our mempool files). First green was 9 scripts; **71** +product, error codes, our mempool files). First green was 9 scripts; **74** unmodified v31.1 scripts `run` now. Remaining growth is claimed wallet-client / P2P / mempool / buried-activation scripts, not the product-never skips (`no-wallet`, prune, v1). `getnetworkhashps` is Core @@ -72,7 +72,7 @@ Owner: [`core-functional.md`](./core-functional.md). | Done | Step | |:----:|------| | [x] | Harness + inventory + nightly / labeled `core-functional` job | -| [x] | 71 unmodified v31.1 scripts `run` (was 9) | +| [x] | 74 unmodified v31.1 scripts `run` (was 9) | | [ ] | Claimed COMPAT-done surface is `run` or an explicit dialect/differ note (**Q-41**) | ### Fuzz until junk input is boring @@ -110,7 +110,7 @@ Store, P2P, RPC, Electrum server, the node — stay in this repo. Revisit ### Faster scripthash, less RAM -Operators feel **wall-clock after IBD** (`--shindex`) and **RSS** while +Operators feel **wall-clock after IBD** (`--sh-index`) and **RSS** while syncing, while building the index, and at tip with wallets connected. Target **2 GiB** process RSS in all three phases; knobs may slow the machine to hit it. Measure on a real SSD. Page cache is not a leak @@ -118,8 +118,8 @@ machine to hit it. Measure on a real SSD. Page cache is not a leak Resume is already the contract: SIGINT keeps sealed SH heads; restart packs unsealed shards only ([`OPERATOR.md`](../OPERATOR.md) § Scripthash). -Pack workers auto-tune to **one per 2 GiB** host free RAM (`RBITCOIN_SH_MERGE_WORKERS` -override). Collect is nCPU. `tx.head` wipe-rebuild is one worker per 1 GiB. +Collect and pack share `sh_extract_workers()`: at most **one per 1.5 GiB** host free RAM (`RBITCOIN_SH_MERGE_WORKERS` +override). `tx.head` wipe-rebuild is one worker per 1 GiB. Operator copy still says post-IBD SH is **tens of minutes to a few hours**; enable-after-tip is often multi-hour. 1.0 needs a named host number that beats 0.5, not a new resume design. @@ -127,7 +127,7 @@ beats 0.5, not a new resume design. | Done | Step | |:----:|------| | [x] | Resume-safe SH materialize (sealed heads kept; unsealed shards only) | -| [x] | Auto-tune SH pack workers: at most **one per 2 GiB** host free RAM (Linux / Darwin / Windows; env still overrides) | +| [x] | Auto-tune SH extract workers: at most **one per 1.5 GiB** host free RAM (collect and pack; Linux / Darwin / Windows; env still overrides) | | [x] | Auto-tune `tx.head` wipe-rebuild: one worker per **1 GiB** free RAM | | [ ] | 2 GiB process RSS in IBD, SH build, and tip-follow (host measure) | | [ ] | Named post-IBD SH wall-clock on laptop SSD, faster than 0.5 | diff --git a/docs/rpc.md b/docs/rpc.md index cc577ce0d..6d9a16cf2 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -55,7 +55,7 @@ later Core RPC is present, including descriptor-wallet calls from 0.21 | `--rpc-listen [ADDR]` / conf `rpc_listen=` | **off** | TCP JSON-RPC; omit ADDR → `127.0.0.1` and Core-matching port (8332 / 18332 / 38332 / 18443). Implies `--rpc`. | | `--rpc-token-file PATH` | `{datadir}/rpc.token` | CSPRNG hex token; TCP `Authorization: Bearer` | | `--rpc-cookie-file PATH` / conf `rpc_cookie_file=` | **off** | Existing Core-format `username:password` file (no trailing newline) accepted as TCP HTTP Basic in addition to Bearer; it is never created or logged | -| `--sh-index` | **off** | Class B scripthash (Electrum/Esplora only; RPC by height/hash/txid does not need it) | +| `--sh-index` | **off** | Class B scripthash for Electrum/Esplora address history and RPC `scantxoutset`. RPC by height/hash/txid does not need it | | `--block-filter-index` | **off** | BIP158 basic. IBD seals them when the flag is on from the start. A later enable still materializes after catch-up. `NODE_COMPACT_FILTERS` is advertised once filters first reach the tip (`getnetworkinfo` lists `COMPACT_FILTERS`), then for the life of the process. `getblockfilter` serves a sealed height, one block whose parent filter is already sealed, and a stale branch back to that sealed fork point. An unsealed best-chain gap is refused. `/rest/blockfilter/` and P2P still serve only heights the watermark covers. Independent of `--sh-index` | | `--rpc-work-queue N` | **16** | In-flight HTTP RPC (Core `-rpcworkqueue`). One POST is one slot (a JSON-RPC array is still one slot). Full permit is HTTP **503** `Work queue depth exceeded`. **0** is the default queue of 16. | | `--rest` / conf `rest=` | **off** | Unauthenticated `/rest/` on the RPC listener. Own queue, same depth as `--rpc-work-queue`'s default. The body is read before that permit is taken. Without the flag those paths are 404. | @@ -116,7 +116,7 @@ rbitcoin-cli --network regtest --rpc-url http://127.0.0.1:18443 getblockchaininf |------------|----------------------|-------------| | IBD, tip follow, P2P, mempool relay | Yes | Yes | | Node JSON-RPC (by height/hash/txid) | Yes | Yes | -| Electrum / Esplora listen | **Refuse start** | Yes after SH tip-ready | +| Electrum / Esplora listen | Start; address methods fail closed (`scripthash index disabled`) | Yes after SH tip-ready | | SH run enqueue / tip bulk | **Skip** | On | Tip-follow readiness is **independent** of scripthash materialize. Electrum/Esplora diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md index 39eb5f085..e34bb5ec6 100644 --- a/docs/sv2-template-provider.md +++ b/docs/sv2-template-provider.md @@ -1,20 +1,20 @@ # SV2 template provider (TDP server) -Roadmap for **Q-64**: three plans, each one PR -([`how-we-plan.md`](./how-we-plan.md#checklist-for-authors-and-agents)). -Cycle and step shape: [agent contract](./how-we-plan.md#agent-contract). Do -not start a plan before the previous one is merged, or a step before the -previous slice is committed. +Design record for **Q-64**. Plans A–C are in tree +([`quality.md`](./quality.md) keeps Q-64 off the open list). The steps +below are how they landed, not work left to start. +Cycle and step shape: [agent contract](./how-we-plan.md#agent-contract). | Plan | Outcome | Ships flags | |------|---------|-------------| | **A** | Landed. GBT and `generate` build from `MempoolHub::select_block_template` | none | -| **B** | A Job Declarator Client mines a block through the node's TP | `--sv2-tp-listen`, `--sv2-tp-authority-sec` / `--sv2-tp-authority-sec-file`, `--sv2-tp-cert-validity`, `--sv2-tp-stale-grace` | -| **C** | Templates refresh on fee gain with the tip unchanged | `--sv2-tp-fee-delta`, `--sv2-tp-template-interval` | +| **B** | Landed. A Job Declarator Client mines a block through the node's TP | `--sv2-tp-listen`, `--sv2-tp-authority-sec` / `--sv2-tp-authority-sec-file`, `--sv2-tp-cert-validity`, `--sv2-tp-stale-grace` | +| **C** | Landed. Templates refresh on fee gain with the tip unchanged | `--sv2-tp-fee-delta`, `--sv2-tp-template-interval` | -B is not split further: a listener that serves templates without +B was not split further: a listener that serves templates without tip-change push or `SubmitSolution` makes miners work stale tips or lose -found blocks. The smallest safe operator surface is B whole. +found blocks. The smallest safe operator surface is B whole. Live flags: +[`OPERATOR.md`](../OPERATOR.md) and [`COMPAT.md`](../COMPAT.md). ## Goal