From b07432f547e1268b3660661099a438cf51512323 Mon Sep 17 00:00:00 2001 From: Brandon Black Date: Thu, 8 Oct 2026 15:04:31 -0700 Subject: [PATCH] docs: match the schema 26 open path and close finding 085 The operator upgrade table still described a rewrite that stopped at schema 25 and grew headers. Open shrinks 24/25 headers to 88 bytes and stamps meta 26. The IBD mutated-body path for finding 085 is already fixed. Co-authored-by: Cursor --- COMPAT.md | 2 +- changelog.d/docs-match-schema-26.md | 8 ++++++++ crates/rbitcoin-primitives/src/lib.rs | 5 +++-- docs/experimental-mainnet.md | 4 ++-- .../085-64-byte-tx-mutation.md | 19 ++++++++----------- docs/external_findings/README.md | 2 +- docs/operator/storage.md | 15 +++++++++------ docs/quality.md | 8 ++++---- docs/road-to-1.0.md | 12 ++++++------ docs/sv2-template-provider.md | 11 ++++++----- 10 files changed, 48 insertions(+), 38 deletions(-) create mode 100644 changelog.d/docs-match-schema-26.md diff --git a/COMPAT.md b/COMPAT.md index c51378ce3d..488aee424b 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -77,7 +77,7 @@ Full `/tx/:txid` JSON still has `vin[]`. Electrum has no outspend-vin surface. | Scripthash index | Optional (`--shindex`, default **off**); bulk at tip when on | External ElectrumX / Fulcrum; Core `-txindex` is different (txid→block) | | JSON-RPC | Documented **subset** ([`docs/rpc.md`](./docs/rpc.md)); Bearer token or opt-in Core cookie (`--rpc-cookie-file`), no `--rpcuser` / `--rpcpassword`; `rbitcoin-cli`. `--rpc-work-queue` defaults to **16** (HTTP occupancy, 503 when full; **0** is that default); a JSON-RPC array is one POST | Full Core RPC; `-rpcworkqueue` is in-flight HTTP jobs (503) | | GetData serve | Every hash is served in order. While **16** (`MAX_SERVE_BLOCKS`) getdata bodies or 4 MiB sit in the peer's queue (RAM cap), the rest waits on the session and no new message is read; none is dropped. Pings and the ping timeout still run | Core `ProcessGetData` pauses on `fPauseSend` and serves the rest before the next message | -| Inbound eviction victim | After Core-shaped protect (netgroup / recent block / recent tx / min-ping), disconnect the **longest-connected** remaining inbound | Core `SelectNodeToEvict` youngest in the oldest netgroup | +| Inbound eviction victim | After Core-shaped protect (netgroup / recent block / recent tx / min-ping / 8 longest-connected), disconnect the **newest** inbound in the largest remaining netgroup. A fully protected set rejects the new inbound | Core `SelectNodeToEvict` youngest in the oldest netgroup | | `--sptweaks-dust` | Serve-time floor default **1000** sat (omit P2TR outs `value <=` floor). **546** matches Cake electrs. Not Cake/Electrum protocol | n/a (Electrum tweaks are not Core) | | IBD empty-headers EOF | Empty `headers` to **our locator** + idle path latches `headers_done` even if a peer advertises a taller less-work height / junk `version.start_height`. A proven header walk at the tip that every taller peer failed to extend also ends IBD | Core header sync follows most-work; advertised `start_height` is not a remaining header count | diff --git a/changelog.d/docs-match-schema-26.md b/changelog.d/docs-match-schema-26.md new file mode 100644 index 0000000000..72b118bdc9 --- /dev/null +++ b/changelog.d/docs-match-schema-26.md @@ -0,0 +1,8 @@ +Fixed + +- **Operator schema upgrade matches schema 26.** Opening a schema 24 or 25 + datadir shrinks `header.body` from 96 B to 88 B and rewrites `meta` to 26. + The operator table no longer describes a rewrite that stops at 25 or grows + headers from 88 B to 96 B. +- **Finding 085 is closed on IBD.** A coinbase-less 64-byte body is a bad + copy: IBD drops it and asks again, and does not cache the hash as invalid. diff --git a/crates/rbitcoin-primitives/src/lib.rs b/crates/rbitcoin-primitives/src/lib.rs index ddba294e46..c71a54a0c0 100644 --- a/crates/rbitcoin-primitives/src/lib.rs +++ b/crates/rbitcoin-primitives/src/lib.rs @@ -105,8 +105,9 @@ pub const SCHEMA_VERSION: u16 = 26; /// /// Schema **26** is current. Meta **22** through **26** opens. Occupied **22** /// rewrites `create.loc.ovf` 12 B rows to 16 B. Occupied **24/25** strips -/// trailing size/weight from 96 B `header.body` rows. Occupied **25** rewrites -/// `meta` and zero-extends `txstat.body`. Meta older than **22** is not +/// trailing size/weight from 96 B `header.body` rows. Any meta below **26** +/// is rewritten to 26; a short `txstat.body` is aligned to the loc count. +/// Meta older than **22** is not /// openable: those Class A bodies (`tx.body`, `txout.body`, `inwit.body` / /// `seqsigwit.body`) were wiped, not rewritten. #[inline] diff --git a/docs/experimental-mainnet.md b/docs/experimental-mainnet.md index 462d0a5c76..ccfb289488 100644 --- a/docs/experimental-mainnet.md +++ b/docs/experimental-mainnet.md @@ -131,7 +131,7 @@ payloads and oversized pending-cmpct pressure; disconnects the peer. **BIP324 v2 only** — discovery is v2-filtered (`x809` DNS + `P2P_V2` gossip); see [`OPERATOR.md`](../OPERATOR.md) § P2P transport. Expect fewer usable -peers than a dual-stack Core node (experimental user-agent still limits inbound). +peers than a dual-stack Core node. The user-agent is `/rbitcoin:VERSION/`, which some peers still refuse. ## Ops risks @@ -139,7 +139,7 @@ peers than a dual-stack Core node (experimental user-agent still limits inbound) |------|--------| | Disk / RAM | Multi‑100 GiB Class A; segmented 25-bit `tx.head.*` + mapped `.fuse8` (~1.5 GiB `RssFile`, heap `fuse8=0`); SH BDZ3 occ mapped (~150 MiB `file=` at ~1 B keys); sealed BDZ `g` FdOnly (not anon heap) | | `tx.head` seal | Segment roll builds fuse8 on seal (~27 M keys); watch seal begin/done logs — not a mono-head shadow fill | -| Peer scarcity | [`OPERATOR.md`](../OPERATOR.md) § P2P transport (`x809` seeds + `P2P_V2` gossip). Experimental user-agent still limits inbound | +| Peer scarcity | [`OPERATOR.md`](../OPERATOR.md) § P2P transport (`x809` seeds + `P2P_V2` gossip). `/rbitcoin:VERSION/` is not a Core user-agent, so some peers still refuse inbound | | Mempool | Libre policy (0.1 sat/vB, full RBF + pure RBFR 1.25×, no dust ban, Libre annex); cluster **64 / 101 kvB**; **scripts verified on accept** | | Confirm lookup/load | **Load** recvs load-sized batches (soft **8000** inputs / hard **144** blocks) from `loadq=14`. Dense mainnet is typically **a few blocks per batch**. IBD **lookup** TipOnly-resolves at most **64000** inputs or **1080** BQ-ready heights per wave, in order from `path_lo`. Real queues loadq=14 · scriptq=4 · writeq=14 | | Not Core/Fulcrum | No production SLA; 0.7.99 is high-scrutiny 0.x; schema unstable until 1.0; occupied 0.6.x stores refuse (wipe + IBD) | diff --git a/docs/external_findings/085-64-byte-tx-mutation.md b/docs/external_findings/085-64-byte-tx-mutation.md index 7237e8e20c..541ea96f1f 100644 --- a/docs/external_findings/085-64-byte-tx-mutation.md +++ b/docs/external_findings/085-64-byte-tx-mutation.md @@ -1,9 +1,7 @@ # 085 — A coinbase-less 64-byte body is mutated **Severity:** high -**Status:** partial — tip, P2P `block`, compact block, and `submitblock` -paths fixed; the IBD confirm path needs the companion change on branch -`consensus/ibd-mutated-body-reject` +**Status:** fixed **Found by:** consensus audit, 2026-10-02 A merkle root also matches a body made of its inner nodes read as @@ -26,16 +24,15 @@ tip hub returns `NetError::Mutated` for any other caller. accept, without a cache. It did cache other mutated bodies (padded witness bytes); it now skips the cache for every `NetError::is_mutated`. -**Scope:** this change covers the tip hub, the P2P `block` handler, -compact reconstruct, and `submitblock`. The IBD body path -(`PeerEvent::BlockFramed` → confirm → `reject_bad_block_tx_layout` → -`ConfirmRejectClass::from_consensus` → `apply_consensus_invalid_reject`) -still marks the hash invalid for this body. The companion change on -branch `consensus/ibd-mutated-body-reject` fixes that path; until it -lands, this finding is not fully fixed. +Block checks compare the merkle root before the other body rules. A first +transaction that is not a coinbase, plus any 64-byte transaction, is +`merkle mutated by a 64-byte tx`. IBD classifies that as a bad copy +(`SoftWire`): the body is dropped, the hash is not cached invalid, and +the block is requested again. **Regression:** `rbitcoin-net` `chain::tests::hostile_peer_session`, `peer::tests::peer_header_dos_and_self_announce`, -`compact::tests::prefilled_64_byte_body_without_coinbase_is_not_a_block`; +`compact::tests::prefilled_64_byte_body_without_coinbase_is_not_a_block`, +`ibd::events::confirm_reject_tests::ibd_mutated_body_is_refetched_not_blacklisted`; `rbitcoin-rpc` `methods::tests::rpc_regtest_from_genesis`. diff --git a/docs/external_findings/README.md b/docs/external_findings/README.md index 3ff53017d3..6565b4c317 100644 --- a/docs/external_findings/README.md +++ b/docs/external_findings/README.md @@ -89,7 +89,7 @@ rbitcoin reference, or redteam static analysis). Numbered reports live beside th | [082](./082-invalid-hash-cap.md) | low | Invalid-hash set stops at 4096 | fixed | `invalid_hash_set_stops_at_the_cap` | | [083](./083-mempool-expiry-cursor.md) | low | Mempool expiry runs without a new admission | fixed | `expire_stale_drops_old_tx_without_a_new_accept` | | [084](./084-findanddelete-empty-sig.md) | critical | Empty sig in legacy CHECKMULTISIG did not delete OP_0 from scriptCode | fixed | `legacy_multisig_empty_sig_deletes_op_0_from_script_code` | -| [085](./085-64-byte-tx-mutation.md) | high | Coinbase-less 64-byte body cached as an invalid block hash | partial (IBD confirm path: `consensus/ibd-mutated-body-reject`) | `hostile_peer_session`, `peer_header_dos_and_self_announce`, `prefilled_64_byte_body_without_coinbase_is_not_a_block`, `rpc_regtest_from_genesis` | +| [085](./085-64-byte-tx-mutation.md) | high | Coinbase-less 64-byte body cached as an invalid block hash | fixed | `hostile_peer_session`, `peer_header_dos_and_self_announce`, `prefilled_64_byte_body_without_coinbase_is_not_a_block`, `ibd_mutated_body_is_refetched_not_blacklisted`, `rpc_regtest_from_genesis` | | [086](./086-p2sh-spk-stack-size.md) | high | P2SH spend skipped the scriptPubKey `MAX_STACK_SIZE` check | fixed | `p2sh_script_pubkey_push_counts_against_max_stack_size` | | [087](./087-forward-spend-in-batch.md) | critical | Spend of a later block in one confirm batch accepted | fixed | `same_batch_spend_of_a_later_block_is_missing`, `batched_spend_of_a_later_block_is_rejected_alone`, `spender_below_its_create_height_is_corrupt` | | [088](./088-tip-chain-selection.md) | high | Tip-follow chain selection: total-work ranking, failed-branch prefix, sibling header check, reorg depth | partial (F3 reorg depth open) | `held_branch_with_more_total_work_beats_an_earlier_local_tie`, `failed_branch_tip_keeps_the_heavier_valid_prefix`, `sibling_claiming_more_work_with_wrong_bits_keeps_the_tip` | diff --git a/docs/operator/storage.md b/docs/operator/storage.md index abc53e3629..64f87d095d 100644 --- a/docs/operator/storage.md +++ b/docs/operator/storage.md @@ -82,14 +82,17 @@ names the dirs. Corrupt files are **not** repaired in-process. | Incoming `meta` | What this binary does | |-----------------|------------------------| -| **25** | Open. Leftover `inwit.*` is renamed to `seqsigwit.*`. Missing `input.loc` with a matching `seqsigwit` count backfills parent edges from those prevouts. Leftover `inputs.*` is renamed to `input.*`. | -| **24** | Rewrite `meta` to 25, then create/extend zeroed `txstat.body` to `create.loc` count (no `txout.body` rewrite). Same `inwit` rename and `input` backfill as 25. Unlink leftover `txfixed.body`. | -| **23** | Rewrite `meta` to 25 first, then rewrite `header.body` 88 B rows to 96 B (size/weight 0) on open. Class A tx stems kept. A torn `header.body` rewrite is retried. Zero-extend `txstat.body`. | -| **22**, occupied Class A | Rewrite `meta` to 25 first, then `create.loc.ovf` 12 B→16 B on `TxTable::open`, then `header.body` 88→96. Crash window is 25 `meta` + old ovf/header; this binary retries those file rewrites. Zero-extend `txstat.body`. | -| **22**, empty Class A | Rewrite `meta` to 25, then open. | +| **26** | Open. | +| **25** | If `header.body` is still 96 B, rewrite each row to 88 B (`header.body.grow`, then rename), dropping trailing size and weight. An 88 B body is left alone. Then rewrite `meta` to 26. | +| **24** | Same header shrink as 25. Zero-extend `txstat.body` to the `create.loc` count (no `txout.body` rewrite). Unlink leftover `txfixed.body`. Rewrite `meta` to 26. | +| **23** | Headers stay 88 B. Rewrite `meta` to 26. `txstat.body` is created and aligned to the loc count when it is missing or short. | +| **22**, occupied Class A | Rewrite `create.loc.ovf` 12 B→16 B. Headers stay 88 B. Then the same `meta` rewrite and `txstat` align as 23. | +| **22**, empty Class A | Rewrite `meta` to 26, then open. | | **Below 22**, empty or occupied | **Refuse.** One line: `schema before 22 refuses this datadir; wipe datadir and redo IBD`. Wipe the datadir and redo IBD. No per-version rewrite. | -A `txstat.body` cell written as four ULEBs starting with `n_in` (an unreleased 25 experiment) is not detected and is not rewritten. Resync that datadir. A **24 binary** refuses 25 `meta` (do not downgrade in place). A **23 binary** refuses 24+ `meta`. A **22 binary** refuses 23+ `meta`. A **21 binary** refuses 22+ `meta`. A **19 binary** refuses 20+ `meta`. +On every open of meta 22–25 the node also renames leftover `inwit.*` to `seqsigwit.*` and `inputs.*` to `input.*`. A missing `input.loc` with seqsigwit prevouts backfills parent edges. A crash during the 96 B→88 B rewrite leaves the old `meta`; the next open retries the shrink. Size and weight after that come from `txstat`. + +A `txstat.body` cell written as four ULEBs starting with `n_in` (an unreleased 25 experiment) is not detected and is not rewritten. Resync that datadir. A **25 binary** refuses 26 `meta`. A **24 binary** refuses 25 `meta` (do not downgrade in place). A **23 binary** refuses 24+ `meta`. A **22 binary** refuses 23+ `meta`. A **21 binary** refuses 22+ `meta`. A **19 binary** refuses 20+ `meta`. When meta is below 22, the log line is: diff --git a/docs/quality.md b/docs/quality.md index 5f289bdf4c..2e10038f64 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -5,7 +5,10 @@ in [`CHANGELOG.md`](../CHANGELOG.md). 1.0 product gates: [`road-to-1.0.md`](./road-to-1.0.md). Peer-node notes: [`peer-clients.md`](./peer-clients.md) (do not copy here). -**Last reaudit:** 2026-09-17. Live schema: [`SCHEMA.md`](../SCHEMA.md). +**Last reaudit:** 2026-10-08. Live schema: [`SCHEMA.md`](../SCHEMA.md) +(`SCHEMA_VERSION = 26`). **Q-64**, **Q-71**, and **Q-72** are off this list: +SV2 plans A–C are in tree, and the Talip and Livera indexes are fully +dispositioned. Core functional inventory: [`core-functional.md`](./core-functional.md). Findings **001–023** fixed. Nightly fuzz **21** jobs. Nightly mutants, 8 hour budget split between new code and the backlog, cursor on the @@ -36,9 +39,6 @@ Counts and procedures stay in the owner doc. A row is the rank and the outcome. | 5 | **Q-67** | `asked_blocks` clone on hold | `hold_body` clones `asked_blocks` before `held_bodies` insert so the read lock does not overlap the write (`HeldBodies::insert` already takes `&HashSet`). Bound is `MAX_SERVE_BLOCKS` × peers. Follow-up: pass the read guard with a documented lock order, or keep the clone as a named trade. Owner: `crates/rbitcoin-net/src/chain.rs`. | | 6 | **Q-69** | CLN / LDK chain backend | Operator can point CLN `bcli` and ldk-node Esplora/Electrum at this node. Owner: [`lightning.md`](./lightning.md). | | 7 | **Q-70** | Batch lookup-path hit counters if a profile names them | `head_resolve_stats::add_hit_rank` does two relaxed `fetch_add`s per resolved txid (`head_resolve_denserels`, `tx_table`). SH extract collect flushes output and hit counts once per fk batch, not per output. Still once per event: `accepted_wb` (`SeqCst`, per confirmed block), `serve_perf::note_serve` (per historical getdata), `page_ios` (per SH page read). Done: a profile shows `add_hit_rank` on confirm lookup and those counters flush per batch, or the profile shows the line is noise and this row moves to Won't-fix. `add_hit_ages` is already a batched flush; `add_hit_age` remains the one-bucket path. | -| 8 | **Q-71** | Talip review remediations | Every row in [`external_findings/024-talip-review-index.md`](./external_findings/024-talip-review-index.md) is fixed, rejected, or won't-fix, and each fixed row names a regression. | -| 9 | **Q-64** | SV2 template provider (TDP server) | Node serves Noise-encrypted Template Distribution Protocol in-process: `CoinbaseOutputConstraints` → pushed `NewTemplate` / `SetNewPrevHash` on tip change and fee delta, `RequestTransactionData`, `SubmitSolution` → `accept_block`. Roadmap (plans A → B → C, one PR each): [`sv2-template-provider.md`](./sv2-template-provider.md). | -| 10 | **Q-72** | Livera review remediations | Every row in [`external_findings/052-livera-review-index.md`](./external_findings/052-livera-review-index.md) is fixed, rejected, or won't-fix, and each fixed row names a regression. | R-ids were the 2026-08-12 slice. Canonical id is **bold**. Do not start **R-11+**. Next unused Q-id is **Q-73**. diff --git a/docs/road-to-1.0.md b/docs/road-to-1.0.md index bac3f40193..ab598c5e75 100644 --- a/docs/road-to-1.0.md +++ b/docs/road-to-1.0.md @@ -46,7 +46,7 @@ taking on for 1.0. `--prune-seqsigwit` is in (seqsigwit-window prune, - Wallet keys, GUI, Core block-file prune (`-prune` of `blk` files), ZMQ, IPC, plaintext v1 P2P - Explorer-search APIs (address-prefix), Liquid, in-binary `/api/v1/` catalogue. **0.8** is electrs HTTP drop-in (except prefix): - [`COMPAT.md`](../COMPAT.md) (**Q-68**) + [`COMPAT.md`](../COMPAT.md) - Every Bitcoin Core functional test (no wallet / prune / v1 scripts) - Matching Core `estimatesmartfee` numbers - Apple notarization @@ -196,9 +196,9 @@ target (cold / after a fee spike) on a real mempool — not a soak badge When the rest is true, tag 1.0 so **every 1.x.x opens a 1.0.0 store**. Older-than-1.0 or corrupt files can still refuse with a one-line message. -Do not freeze while Class C / sidecar / fuse8 can lose a `set` or index -OOB (**Q-57**). Mempool persist writes body before LIVE slots (**Q-58**). -Schema 21 is the current bytes; 0.x may still bump. +The durable path is fail-closed (**Q-57** / **Q-58**). Live bytes are +schema **26** ([`SCHEMA.md`](../SCHEMA.md)). 0.x may still bump; 1.0 is +the freeze. | Done | Step | |:----:|------| @@ -215,11 +215,11 @@ mempool.space Node+MariaDB+frontend can retire electrs. Core JSON-RPC for that stack is TCP plus an opt-in Core cookie (`--rpc-cookie-file`), so stock mempool `CORE_RPC` works unpatched; a unix-socket patch stays optional. Not their `/api/v1/` process. Not address-prefix. -**Q-68**. Surface: [`COMPAT.md`](../COMPAT.md). +Surface: [`COMPAT.md`](../COMPAT.md). (**Q-68** is the create.loc SIMD +extract in [`TESTING.md`](../TESTING.md), not this drop-in.) | Done | Step | |:----:|------| -| [x] | Q-68 parked (this row) | | [x] | Esplora unix + `/internal/*` + mempool tx-JSON snapshot | | [x] | Product docs claim the drop-in | diff --git a/docs/sv2-template-provider.md b/docs/sv2-template-provider.md index 6fc0dc7078..39eb5f0854 100644 --- a/docs/sv2-template-provider.md +++ b/docs/sv2-template-provider.md @@ -591,10 +591,11 @@ No live pool/JDC, no mainnet datadir, no plaintext mode. `bad-diffbits` unless the sent bits were already the limit. A re-push cannot fix it: a same-hash `SetNewPrevHash` is outside §7.4, and a `NewTemplate` has no field for the bits. Not planned; mainnet and signet bits depend only on the prev hash. -- Pre-existing, outside this plan: consensus does not enforce the BIP94 - timewarp floor (testnet4). The first block of a retarget period may carry - a timestamp more than 600 s before its parent. Core rejects it - (`time-timewarp-attack`); when consensus gains the rule, the template - timestamp floor must include it. +- Pre-existing, outside this plan: this binary has no testnet4 network. + If one is added, consensus does not yet enforce the BIP94 timewarp floor. + The first block of a retarget period may carry a timestamp more than 600 s + before its parent. Core rejects that (`time-timewarp-attack`); the template + timestamp floor must include the rule when consensus gains it. Mainnet, + testnet3, signet, and regtest do not use that rule. - After ship: JD-server mode, weak blocks, extension negotiation, per-IP connection limits → quality.md rows, not this roadmap.