diff --git a/CHANGELOG.md b/CHANGELOG.md index d031685a5..746556ece 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,34 @@ before 1.0). ### Changed +- **Pruned SH materialize is two-pass extract:** each collect worker owns a + contiguous create-fk span and unsized maps capped at 1.5 GiB + (`SH_EXTRACT_WORKER_RAM_BYTES`; 64 B/key pass 1, `80n+8f` pass 2). After + each 64 k-fk loc/body batch, spill the largest shard map while over + budget (`SHKSP01` under `keys/NN/`, first-fk delta singles). Collect spills + are tmp+rename without `sync_all` (no `DONE.keys` still wipes unsorted). + Merge folds those spills into one map, one walk to pack8 + `scripthash.head/NN` (singles `inline_one`, multis Empty; file exists is + not pack-done) and fuse8 of dupes to `multi/NN.fuse8` (on disk so other + shards do not keep it resident), then unlinks `keys/NN/`. The folded + map is consumed into the pack8 records and dropped before fuse8 and BDZ. + Pass 2 keeps fuse8 only (no BDZ): same static spans; fuse-hit creates + fold into per-worker `key16 → Vec` maps and spill-largest as + `SHPST01` under `post/NN/`. A `post/NN` file, or a spill whose magic is + not `SHPST01`, is Corrupt. Pack folds those spills into one map + (~0.3–0.5 GiB/shard; a few GiB for 8 workers), then `slot_for_key16` + + 2+ bodies (grouped by MPHF slot); `len == 1` after fold is `fp_singles`. `DONE.post` is + `SHPOST02` last_fk. Keys already unlinked when Class A grows before + pack: full recollect (MPHF tags are not key16). All extract phases + share `sh_extract_workers()` = min(CPUs, max(1, free RAM / 1.5 GiB)); + `RBITCOIN_SH_MERGE_WORKERS` still overrides. Collect maps are the + 1.5 GiB worker cap (spilled and dropped before merge BDZ). Progress is + `scanned=` finished fks. Output and hit counters flush once per 64 k-fk + batch. Spills share one writer (1-slot queue). One + `keys merge start`; live `keys merge shard=` with `fold=` `bdz=`; pack + shard lines when each worker finishes. Previous `DONE` / 24 B `NN` + unsorted is deleted and pass 1 restarts. + - **PR cargo-mutants is 4 in-diff shards:** `ci.yml` `mutants`, after fmt/clippy/test, in parallel with coverage. Advisory (`continue-on-error`), not a merge gate. One 30-minute job was canceled diff --git a/OPERATOR.md b/OPERATOR.md index f73267131..c7793e9f0 100644 --- a/OPERATOR.md +++ b/OPERATOR.md @@ -895,13 +895,23 @@ Tip-follow readiness is **independent** of SH materialize (`tip_follow_ready` ### Abort / resume (tip materialize) -Keep **`store/scripthash.unsorted/`** until all shards seal. SIGINT / SIGTERM -mid-cold keeps every **sealed** `scripthash.head/NN`; restart with the same +Keep **`store/scripthash.unsorted/`** until all shards seal. Extra disk during +build is **`SHKSP01` spills** (one rec per unique key per worker map) plus +**`SHPST01` post spills** (one rec per unique multi key per map, delta fks). +SIGINT / SIGTERM +mid-cold keeps every **RAM-published** `scripthash.head/NN`; restart with the same `--datadir --sh-index` packs **unsealed** shards only (holes stay). Incomplete -collect (no `DONE`) restarts the Class A pass. `DONE` names the Class A -`create_fk` scanned; restart appends new creates into unsorted files when no +pass 1 (no `DONE.keys`) restarts the first Class A scan. A previous layout +(`DONE` / 24 B `NN` files, or `keys/NN` / `post/NN` as a file) with no valid +`DONE.keys` is deleted and pass 1 starts over. A spill whose magic is not +`SHKSP01` or `SHPST01` is Corrupt — wipe `store/scripthash.unsorted` and +rematerialize. `DONE.keys` / `DONE.post` name the Class A +`create_fk` scanned; restart appends new creates when no shards are sealed, or tail-appends onto the durable head after pack when any -`head/NN` already exists. Do not delete unsorted files +`head/NN` is already published. Extract phases (collect, merge, BDZ, fuse, +pass 2, pack) share one worker cap (`store: scripthash … workers=`; +`RBITCOIN_SH_MERGE_WORKERS` override). Do not +delete unsorted files to “start over” unless you intend a full Class A collect (`RBITCOIN_SH_FORCE_REBUILD`). Leftover `scripthash.runs` are discarded at tip (never k-way rematerialized). @@ -910,8 +920,8 @@ to “start over” unless you intend a full Class A collect |------|-------------------| | SIGTERM / SIGINT mid pack | Resume. Sealed `head/NN` stays; unsealed shards re-pack from unsorted files. | | Kill-9 mid pack | Same idea; unfinished shard work is redone. Open follows [`docs/crash-recovery.md`](docs/crash-recovery.md) (scripthash Direct). | -| `DONE` then more Class A, no sealed shards | Append the new fk span into unsorted files, then pack. | -| `DONE` then more Class A, some/all shards sealed | Pack remaining unsealed files; Class A tail onto the durable head (Direct) or write-behind (Tip). | +| `DONE.keys` / `DONE.post` then more Class A, no sealed shards | Append the new fk span into keys then postings, then pack. | +| `DONE.keys` / `DONE.post` then more Class A, some/all shards sealed | Pack remaining unsealed `post/NN`; Class A tail onto the durable head (Direct) or write-behind (Tip). | | Empty SH head + leftover catalog | Wipe leftover runs + SEAL, then Class A collect into unsorted shards. | | Durable SH head + leftover runs | Discard leftover runs (keep SEAL); write-behind fills HWM lag. | | Corrupt SH (leftover live OA, mixed body, refuse line) | Wipe `store/scripthash*` only, keep Class A, rematerialize with `--sh-index`. | diff --git a/SCHEMA.md b/SCHEMA.md index 0715eb48c..96a43452c 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -227,7 +227,10 @@ itself changed. scripthash.ovf/NNNNNN[.fuse8][.idx] # L0 SHSR pack8 scripthash.ovf/NNNNNN.mphf|.val|.fuse8 # L1 promoted ovf (at most one) scripthash.runs # leftover catalog (key_len=40); discarded at tip - scripthash.unsorted/NN # tip collect: raw 24 B recs (prefix16+fk), unsorted; DONE=SHUNSRT3+last_fk+counts; unlinked after seal + scripthash.unsorted/keys/NN/ # pass 1 dir of SHKSP01 spills (000000…); merge identity-map fold + one walk to head+multi fuse, unlinks; DONE.keys=SHKEYS02 last_fk marker + scripthash.unsorted/multi/NN.fuse8 # throwaway fuse8 of 0xFFFF keys for pass 2 + scripthash.unsorted/post/NN/ # pass 2 dir of SHPST01 spills (000000…); pack folds to one map then slot_for_key16 + 2+ body; DONE.post=SHPOST02 last_fk; unlinked after pack + # previous DONE / 24 B NN (SHUNSRT3) with no SHKEYS02 is wiped and pass 1 restarts; no SCHEMA_VERSION bump sp_tweaks.idx/ sp_tweaks.body/ # optional BIP-352 (schema 17 dirs; leftover files unlinked) / # only when --datadir-cold is set @@ -699,8 +702,17 @@ compact still merges **heads only** — all ovf keys share ### Query join Heights, value, spentness, vouts: expand from Class A outputs (match full scripthash) + spend annotations + Class C. -IBD may stage creates in **unsorted per-shard files** (24 B `{sh_prefix16\|create_fk}`) -and unique-sort + pack durable SH at tip entry. Leftover schema-16 `key_len=32` +IBD may stage creates in **two Class A `txout` scans** under `scripthash.unsorted/`: +workers write `SHKSP01` spill files under `keys/NN/` +(n_multi key16s already `0`, then first-fk-sorted uleb(delta)‖key16 singles), +then one map-fold walk per shard into pack8 `scripthash.head/NN` and +`multi/NN.fuse8` (`DONE.keys` = `SHKEYS02` last_fk marker). A previous +`DONE` / 24 B `NN` layout with no valid `DONE.keys` is deleted and pass 1 +restarts. A spill whose magic is not `SHKSP01` is Corrupt — wipe +`scripthash.unsorted` and rematerialize. Fuse-hit `SHPST01` post spills +under `post/NN/` (`DONE.post` = `SHPOST02` last_fk). A `post/NN` file, or a +spill whose magic is not `SHPST01`, is Corrupt. Pack folds those spills, then +rewrites 2+ into body and skips 1-fk fuse FPs. Leftover schema-16 `key_len=32` catalogs are refused. **Decision:** inline for 1-use scripts (`SH_INLINE_CAP = 1`, ~95 % of keys); geometric slabs for diff --git a/crates/rbitcoin-query/src/catchup.rs b/crates/rbitcoin-query/src/catchup.rs index 025f33bbd..6ad045c32 100644 --- a/crates/rbitcoin-query/src/catchup.rs +++ b/crates/rbitcoin-query/src/catchup.rs @@ -860,7 +860,10 @@ mod tests { let done_last = rbitcoin_store::unsorted_done_last_fk(&udir, n_shards).unwrap(); extend_direct_chain(&q, 2); assert!(q.store.txs.count() > done_last); - assert!(!q.store.scripthash.has_durable_index()); + assert!( + !q.store.scripthash.unsealed_main_shards().is_empty(), + "RAM merge writes head files; pack seal is later" + ); let _ = q.finalize_sh_runs().unwrap(); let sh_old = rbitcoin_store::script_hash(&[0x51, 0]); let sh_new = rbitcoin_store::script_hash(&[0x51, 3]); diff --git a/crates/rbitcoin-query/src/sh_builder.rs b/crates/rbitcoin-query/src/sh_builder.rs index 479080a5c..ae412f6f1 100644 --- a/crates/rbitcoin-query/src/sh_builder.rs +++ b/crates/rbitcoin-query/src/sh_builder.rs @@ -1,5 +1,5 @@ -//! Post-IBD scripthash collect: one Class A pass into unsorted per-shard -//! files, then in-place unique-sort + seal. +//! Post-IBD scripthash collect: two Class A `txout` scans (unique key16, +//! then fuse-hit postings), then pack + seal. //! //! Direct confirm does **not** enqueue SH. A durable head never enters this //! path (write-behind / `recover_sh_writebehind` instead). Leftover diff --git a/crates/rbitcoin-store/src/file.rs b/crates/rbitcoin-store/src/file.rs index 467a7d24b..216a07b0e 100644 --- a/crates/rbitcoin-store/src/file.rs +++ b/crates/rbitcoin-store/src/file.rs @@ -54,9 +54,16 @@ pub(crate) fn tmp_sidecar_path(path: &Path) -> PathBuf { } } +/// Write `bytes` to a sibling tmp, then rename over `path`. No `sync_all`. +pub(crate) fn write_tmp_rename(path: &Path, bytes: &[u8]) -> Result<(), StoreError> { + write_tmp_file(path, false, |f| { + f.write_all(bytes).map_err(|e| StoreError::io(path, e)) + }) +} + /// Write `bytes` to a sibling tmp, `sync_all`, then rename over `path`. pub(crate) fn write_synced_tmp_rename(path: &Path, bytes: &[u8]) -> Result<(), StoreError> { - write_synced_tmp_file(path, |f| { + write_tmp_file(path, true, |f| { f.write_all(bytes).map_err(|e| StoreError::io(path, e)) }) } @@ -65,6 +72,13 @@ pub(crate) fn write_synced_tmp_rename(path: &Path, bytes: &[u8]) -> Result<(), S /// /// The durable name is not created or truncated until rename. pub(crate) fn write_synced_tmp_file(path: &Path, write: F) -> Result<(), StoreError> +where + F: FnOnce(&mut File) -> Result<(), StoreError>, +{ + write_tmp_file(path, true, write) +} + +fn write_tmp_file(path: &Path, sync: bool, write: F) -> Result<(), StoreError> where F: FnOnce(&mut File) -> Result<(), StoreError>, { @@ -77,7 +91,9 @@ where { let mut f = File::create(&tmp).map_err(|e| StoreError::io(&tmp, e))?; write(&mut f)?; - f.sync_all().map_err(|e| StoreError::io(&tmp, e))?; + if sync { + f.sync_all().map_err(|e| StoreError::io(&tmp, e))?; + } } std::fs::rename(&tmp, path).map_err(|e| StoreError::io(path, e))?; Ok(()) @@ -1410,4 +1426,21 @@ mod advise_tests { assert_eq!(std::fs::read(&dest).unwrap(), b"sealed"); let _ = std::fs::remove_dir_all(&dir); } + + #[test] + fn write_tmp_rename_installs_dest_only() { + static N: AtomicU64 = AtomicU64::new(0); + let id = N.fetch_add(1, Ordering::Relaxed); + let dir = std::env::temp_dir().join(format!("rbitcoin-tmp-rename-{id}")); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let dest = dir.join("spill.bin"); + let tmp = tmp_sidecar_path(&dest); + assert!(!dest.exists()); + write_tmp_rename(&dest, b"spill").unwrap(); + assert!(dest.exists()); + assert!(!tmp.exists()); + assert_eq!(std::fs::read(&dest).unwrap(), b"spill"); + let _ = std::fs::remove_dir_all(&dir); + } } diff --git a/crates/rbitcoin-store/src/scripthash.rs b/crates/rbitcoin-store/src/scripthash.rs index 213949166..c643b0b29 100644 --- a/crates/rbitcoin-store/src/scripthash.rs +++ b/crates/rbitcoin-store/src/scripthash.rs @@ -390,7 +390,7 @@ fn file_starts_with_shsr(path: &Path) -> bool { matches!(f.read_exact(&mut magic), Ok(())) && magic == *b"SHSR" } -fn sorted_main_shard_path(dir: &Path, shard: usize, n_shards: usize) -> PathBuf { +pub(crate) fn sorted_main_shard_path(dir: &Path, shard: usize, n_shards: usize) -> PathBuf { let p = dir.join("scripthash.head"); if n_shards <= 1 && p.is_file() { p @@ -2499,6 +2499,7 @@ impl ScriptHashTable { resume_from_shard: 0, active_shard: None, recs: Vec::new(), + slot_words: Vec::new(), key_budget, body_buf: Vec::with_capacity(BULK_BODY_FLUSH), body_write_off: bump, @@ -2547,6 +2548,7 @@ impl ScriptHashTable { resume_from_shard: 0, active_shard: Some(shard), recs: Vec::new(), + slot_words: Vec::new(), key_budget, body_buf: Vec::with_capacity(BULK_BODY_FLUSH), body_write_off: bump, @@ -2609,6 +2611,7 @@ impl ScriptHashTable { resume_from_shard: progress.next_shard, active_shard: None, recs: Vec::new(), + slot_words: Vec::new(), key_budget, body_buf: Vec::with_capacity(BULK_BODY_FLUSH), body_write_off: bump, @@ -2638,6 +2641,18 @@ impl ScriptHashTable { self.allocs[0].lock().unwrap().bump } + /// Pass-1 BDZ singles already inline in `head/NN` (not yet packed body). + pub(crate) fn set_extract_inline_creates( + &self, + shard: usize, + n: u64, + ) -> Result<(), StoreError> { + let body = self.shard_body(shard); + let mut g = self.shard_alloc(shard).lock().unwrap(); + g.live_count = n; + write_alloc_header(body, &g) + } + /// Seal `recs` as sorted main shard `shard` and publish alloc HWM. pub fn publish_sorted_shard( &self, @@ -2683,7 +2698,15 @@ impl ScriptHashTable { if let Some(parent) = path.parent() { let _ = std::fs::create_dir_all(parent); } - let sealed = MphfHead::write_pack8(&path, &pack.recs)?; + let sealed = if MphfHead::exists(&path) { + let h = MphfHead::open(&path)?; + if !pack.slot_words.is_empty() { + h.rewrite_val_slots(&pack.slot_words)?; + } + h + } else { + MphfHead::write_pack8(&path, &pack.recs)? + }; self.install_sorted_main(shard, sealed); let body = self.shard_body(shard); if bump > body.logical_len() { @@ -2725,6 +2748,8 @@ pub struct ScriptHashBulkSession<'a> { active_shard: Option, /// Packed recs for [`Self::bulk_session`] (sorted at shard seal; 16 B key order). recs: Vec<(crate::scripthash_layout::ShHeadKey, u64)>, + /// MPHF slot + pack8 for pass-2 body writes (grouped by slot). + slot_words: Vec<(u32, u64)>, /// Unique-key budget (log / tests). Does not pre-size an OA table. key_budget: u64, /// Sequential slab bytes; flushed at [`BULK_BODY_FLUSH`] or before a @@ -2755,17 +2780,26 @@ pub struct ScriptHashBulkSession<'a> { /// One shard packed onto its live body, ready for ordered head publish. pub struct ShShardPack { pub recs: Vec<(crate::scripthash_layout::ShHeadKey, u64)>, + pub slot_words: Vec<(u32, u64)>, pub creates: u64, pub max_fk: u64, pub keys: u64, pub bump: u64, pub body_flush_ns: u64, pub pack_ns: u64, + /// Fuse-hit postings with one unique create_fk (left inline; no body). + pub fp_singles: u64, } /// One unfinished key in [`ScriptHashBulkSession`] (≤ one delta page of FKs). +#[derive(Clone, Copy)] +enum BulkGroup { + Script([u8; 32]), + Slot(u32), +} + struct BulkOpenKey { - key: [u8; 32], + group: BulkGroup, buf: Vec, stream_used: usize, n_total: u32, @@ -2875,16 +2909,18 @@ impl<'a> ScriptHashBulkSession<'a> { } self.finish_key()?; self.flush_body()?; - let persist_live = self.live_count; + let persist_live = self.committed_live_count.saturating_add(self.live_count); self.persist_session_alloc(persist_live, self.bump)?; let pack = ShShardPack { recs: std::mem::take(&mut self.recs), + slot_words: std::mem::take(&mut self.slot_words), creates: self.live_count, max_fk: self.max_fk, keys: self.keys_written, bump: self.bump, body_flush_ns: self.body_flush_ns, pack_ns: self.pack_ns, + fp_singles: 0, }; self.finished = true; Ok(pack) @@ -2899,7 +2935,11 @@ impl<'a> ScriptHashBulkSession<'a> { if fk.is_null() { return Ok(()); } - if self.open_key.as_ref().is_some_and(|o| o.key != key) { + if self + .open_key + .as_ref() + .is_some_and(|o| !matches!(o.group, BulkGroup::Script(k) if k == key)) + { self.finish_key()?; } if self.open_key.is_none() { @@ -2908,7 +2948,7 @@ impl<'a> ScriptHashBulkSession<'a> { } let buf = self.take_fk_scratch(); self.open_key = Some(BulkOpenKey { - key, + group: BulkGroup::Script(key), buf, stream_used: 0, n_total: 0, @@ -2916,6 +2956,10 @@ impl<'a> ScriptHashBulkSession<'a> { last_fk: None, }); } + self.push_open_fk(fk) + } + + fn push_open_fk(&mut self, fk: Fk) -> Result<(), StoreError> { let add = { let open = self .open_key @@ -2959,6 +3003,37 @@ impl<'a> ScriptHashBulkSession<'a> { Ok(()) } + /// Stream one create_fk for MPHF `slot` (pass-2 pack; grouped by slot). + pub fn push_sorted_slot_fk(&mut self, slot: u32, fk: Fk) -> Result<(), StoreError> { + if fk.is_null() { + return Ok(()); + } + if !self.pack_only { + return Err(StoreError::Corrupt( + "scripthash pack slot stream requires pack_shard_session", + )); + } + if self + .open_key + .as_ref() + .is_some_and(|o| !matches!(o.group, BulkGroup::Slot(s) if s == slot)) + { + self.finish_key()?; + } + if self.open_key.is_none() { + let buf = self.take_fk_scratch(); + self.open_key = Some(BulkOpenKey { + group: BulkGroup::Slot(slot), + buf, + stream_used: 0, + n_total: 0, + first_page: None, + last_fk: None, + }); + } + self.push_open_fk(fk) + } + /// Seal the open key (inline / slab / last page). pub fn finish_key(&mut self) -> Result<(), StoreError> { if self.open_key.is_none() { @@ -3014,11 +3089,18 @@ impl<'a> ScriptHashBulkSession<'a> { }; self.live_count = self.live_count.saturating_add(u64::from(n)); self.keys_written = self.keys_written.saturating_add(1); - let rec = (head_key_from_full(&open.key), pack8(&val)?); - self.recs.push(rec); - self.peak_table_bytes = self - .peak_table_bytes - .max(self.recs.len().saturating_mul(24)); + let word = pack8(&val)?; + match open.group { + BulkGroup::Script(key) => { + self.recs.push((head_key_from_full(&key), word)); + self.peak_table_bytes = self + .peak_table_bytes + .max(self.recs.len().saturating_mul(24)); + } + BulkGroup::Slot(slot) => { + self.slot_words.push((slot, word)); + } + } self.return_fk_scratch(open.buf); Ok(()) } diff --git a/crates/rbitcoin-store/src/scripthash_materialize.rs b/crates/rbitcoin-store/src/scripthash_materialize.rs index 0ac8e25ec..e76ae207e 100644 --- a/crates/rbitcoin-store/src/scripthash_materialize.rs +++ b/crates/rbitcoin-store/src/scripthash_materialize.rs @@ -1,27 +1,72 @@ -//! Unsorted-shard SH tip materialize: one Class A pass into per-shard files, -//! then in-place unique-sort + pack + seal `head/NN`. +//! Two-scan SH tip extract: unique key16 per shard, then fuse-hit postings. //! -//! Sharded bodies: each pack worker writes `body/NN` and seals `head/NN` itself. +//! Pass 1: each worker owns a contiguous create-fk span and `n_shards` unsized +//! identity maps (`key16 → pack8 word`; `0` = multi). Cap is +//! [`SH_EXTRACT_WORKER_RAM_BYTES`] (1.5 GiB estimate, 64 B/key). After each +//! 64 k-fk loc/body batch, spill the largest shard map while over budget +//! (`SHKSP01` under `keys/NN/`, one writer, 1-slot queue). Status is +//! `scanned=` finished fks. Merge folds those spill files into one map, +//! one walk to `scripthash.head/NN` and `multi/NN.fuse8`, then unlinks +//! `keys/NN/`. A `keys/NN` file with no valid `DONE.keys` is wiped with the +//! previous layout. A spill whose magic is not `SHKSP01` refuses. +//! Pass 2 keeps fuse8 only (no BDZ): same static spans and per-worker maps +//! (`key16 → Vec`, estimate `80n+8f`), spill-largest as `SHPST01` under +//! `post/NN/`. Pack folds those spills, then `slot_for_key16` + 2+ bodies; +//! 1-fk after fold is `fp_singles`. A `post/NN` file refuses. use crate::error::StoreError; -use crate::io_handle::IoHandle; -use crate::scripthash::{ColdProgress, ScriptHashTable, ShShardPack}; +use crate::fuse8_filter::SealedFuse8; +use crate::scripthash::{sorted_main_shard_path, ColdProgress, ScriptHashTable, ShShardPack}; use crate::scripthash_head::prefix_shard_of; -use crate::scripthash_layout::SH_HEAD_KEY_LEN; +use crate::scripthash_layout::{head_key_from_full, ShHeadKey, SH_HEAD_KEY_LEN}; +use crate::scripthash_mphf::{mix_key16, mphf_path, val_path, MphfHead}; use crate::store::Store; use crate::tx_table::TxTable; -use rbitcoin_primitives::Fk; -use std::collections::VecDeque; -use std::fs::{self, File, OpenOptions}; -use std::io; +use rbitcoin_primitives::{read_uleb128, write_uleb128, Fk}; +use std::collections::hash_map::Entry; +use std::collections::{HashMap, HashSet, VecDeque}; +use std::fs::{self, OpenOptions}; +use std::hash::{BuildHasherDefault, Hasher}; use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering}; +use std::sync::mpsc::{self, Receiver, SyncSender}; use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; const MATERIALIZE_STATUS_INTERVAL: Duration = Duration::from_secs(10); -/// Global pack/publish counters (per-shard seal logs sample these). +pub const UNSORTED_SHARD_DIR: &str = "scripthash.unsorted"; +const KEYS_SUBDIR: &str = "keys"; +const POST_SUBDIR: &str = "post"; +const MULTI_SUBDIR: &str = "multi"; +#[cfg(test)] +const MPHF_SUBDIR: &str = "mphf"; +const KEYS_DONE_NAME: &str = "DONE.keys"; +const POST_DONE_NAME: &str = "DONE.post"; +const KEYS_DONE_MAGIC: &[u8; 8] = b"SHKEYS02"; +const POST_DONE_MAGIC: &[u8; 8] = b"SHPOST02"; +const LEGACY_DONE_NAME: &str = "DONE"; + +const KEY16_LEN: usize = SH_HEAD_KEY_LEN; +/// One keys-body codec. Anything else is leftover (refuse, do not open). +const KEYS_SPILL_MAGIC: &[u8; 8] = b"SHKSP01\0"; +const KEYS_SPILL_HDR: usize = 16; +/// Collect-map estimate: `64 × n_keys` (HashMap overhead, not a prefault). +const KEYS_COLLECT_BYTES_PER_KEY: u64 = 64; +const INDEX_REFUSE_KEYS_LEFTOVER: &str = "scripthash unsorted keys leftover from an older extract; wipe store/scripthash.unsorted and rematerialize"; +const INDEX_REFUSE_POST_LEFTOVER: &str = "scripthash unsorted post leftover from an older extract; wipe store/scripthash.unsorted and rematerialize"; +/// Inner loc/body batch inside a static worker fk span. +pub(crate) const CLASS_A_CHUNK_FKS: u64 = 1 << 16; +/// One posts-body codec. Anything else is leftover (refuse, do not open). +const POSTS_SPILL_MAGIC: &[u8; 8] = b"SHPST01\0"; +const POSTS_SPILL_HDR: usize = 12; +/// Collect-map estimate: `80 × n_keys + 8 × n_fks`. +const POST_MAP_KEY_BYTES: usize = 80; +const POST_MAP_FK_BYTES: usize = 8; + +/// One extract worker: BDZ `g` + fuse + maps (~1.5 GiB). +pub const SH_EXTRACT_WORKER_RAM_BYTES: u64 = 3 << 29; + struct MaterializeProgress { recs_packed: AtomicU64, keys_packed: AtomicU64, @@ -57,7 +102,6 @@ impl MaterializeProgress { } } -/// CPU-summed stage times for unsorted SH pack (parallel workers add). #[derive(Debug, Clone, Copy, Default)] pub struct MaterializeStageNs { pub merge_ns: u64, @@ -66,7 +110,6 @@ pub struct MaterializeStageNs { pub body_flush_ns: u64, } -/// Result of unsorted SH collect + pack. #[derive(Debug, Clone, Copy)] pub struct ShShardMaterialize { pub creates: u64, @@ -90,108 +133,53 @@ impl ShShardMaterialize { } } -fn decode_unsorted_rec(rec: &[u8]) -> Result<([u8; 32], Fk), StoreError> { - if rec.len() != UNSORTED_REC_LEN { - return Err(StoreError::Corrupt("scripthash unsorted rec length")); +fn check_cancel(cancel: Option<&AtomicBool>, what: &'static str) -> Result<(), StoreError> { + if cancel.map(|c| c.load(Ordering::Relaxed)).unwrap_or(false) { + Err(StoreError::Cancelled(what)) + } else { + Ok(()) } - let mut sh = [0u8; 32]; - sh[..SH_HEAD_KEY_LEN].copy_from_slice(&rec[..SH_HEAD_KEY_LEN]); - let fk = Fk(u64::from_le_bytes( - rec[SH_HEAD_KEY_LEN..].try_into().unwrap(), - )); - Ok((sh, fk)) } -fn seal_shard( - table: &ScriptHashTable, - shard: usize, - pack: ShShardPack, - max_fk: &AtomicU64, - progress: &MaterializeProgress, -) -> Result<(), StoreError> { - max_fk.fetch_max(pack.max_fk, Ordering::Relaxed); - let creates = pack.creates; - let t_mphf = Instant::now(); - table.publish_packed_shard(shard, pack)?; - progress - .mphf_ns - .fetch_add(t_mphf.elapsed().as_nanos() as u64, Ordering::Relaxed); - progress - .creates_published - .fetch_add(creates, Ordering::Relaxed); - progress.shards_published.fetch_add(1, Ordering::Relaxed); - table.store_sharded_cold_progress( - progress.keys_packed.load(Ordering::Relaxed), - progress.creates_published.load(Ordering::Relaxed), - )?; - Ok(()) +pub fn unsorted_shard_dir(store_dir: &Path) -> PathBuf { + store_dir.join(UNSORTED_SHARD_DIR) } -struct ShardPool { - jobs: Mutex>, - err: Mutex>, +pub fn unsorted_shard_path(dir: &Path, shard: usize) -> PathBuf { + dir.join(format!("{shard:02x}")) } -/// Temp dir for one Class A pass → unsorted per-shard 24-byte records -/// (16-byte head prefix + create_fk). -pub const UNSORTED_SHARD_DIR: &str = "scripthash.unsorted"; -const UNSORTED_DONE_NAME: &str = "DONE"; -const UNSORTED_DONE_MAGIC: &[u8; 8] = b"SHUNSRT3"; -const UNSORTED_REC_LEN: usize = SH_HEAD_KEY_LEN + 8; -const UNSORTED_FLUSH_BYTES: usize = 1024 * 1024; -const UNSORTED_ALLOC_STEP: u64 = 64 << 20; -const CLASS_A_CHUNK_FKS: u64 = 64_000; - -/// RAM budget per unsorted-shard pack worker (one file image + unique pack recs). -pub const SH_UNSORTED_PACK_RAM_BYTES: u64 = 2 << 30; - -/// Collect / pack result counts for one unsorted-shard cold pass. -#[derive(Clone, Debug, Default)] -pub struct UnsortedShardCollect { - pub recs: u64, - pub per_shard: Vec, - /// Inclusive Class A create_fk scanned into these files (`txs.count()` at collect). - pub last_fk: u64, +pub(crate) fn unsorted_keys_path(dir: &Path, shard: usize) -> PathBuf { + dir.join(KEYS_SUBDIR).join(format!("{shard:02x}")) } -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum UnsortedCollectAction { - Full, - Append { first: u64, last: u64 }, - Skip, +pub(crate) fn unsorted_keys_spill_path(dir: &Path, shard: usize, seq: u32) -> PathBuf { + unsorted_keys_path(dir, shard).join(format!("{seq:06}")) } -fn plan_unsorted_collect( - done_last: Option, - class_a_last: u64, - any_sealed_shards: bool, -) -> UnsortedCollectAction { - match done_last { - None => UnsortedCollectAction::Full, - Some(d) if d >= class_a_last => UnsortedCollectAction::Skip, - Some(_) if any_sealed_shards => UnsortedCollectAction::Skip, - Some(d) => UnsortedCollectAction::Append { - first: d.saturating_add(1).max(1), - last: class_a_last, - }, - } +pub(crate) fn unsorted_post_path(dir: &Path, shard: usize) -> PathBuf { + dir.join(POST_SUBDIR).join(format!("{shard:02x}")) } -pub fn unsorted_shard_dir(store_dir: &Path) -> PathBuf { - store_dir.join(UNSORTED_SHARD_DIR) +pub(crate) fn unsorted_post_spill_path(dir: &Path, shard: usize, seq: u32) -> PathBuf { + unsorted_post_path(dir, shard).join(format!("{seq:06}")) } -pub fn unsorted_shard_path(dir: &Path, shard: usize) -> PathBuf { - dir.join(format!("{shard:02x}")) +pub(crate) fn unsorted_multi_fuse_path(dir: &Path, shard: usize) -> PathBuf { + dir.join(MULTI_SUBDIR).join(format!("{shard:02x}.fuse8")) +} + +#[cfg(test)] +pub(crate) fn unsorted_mphf_base(dir: &Path, shard: usize) -> PathBuf { + dir.join(MPHF_SUBDIR).join(format!("{shard:02x}")) } -/// Collect workers: nCPU (1 MiB write buffers; no RAM cap / env). pub fn unsorted_collect_workers() -> usize { - crate::sorted_run::logical_cpus() + sh_extract_workers() } -/// Pack workers: `RBITCOIN_SH_MERGE_WORKERS` or free RAM / 2 GiB. -pub fn unsorted_pack_workers() -> usize { +/// `min(nCPU, max(1, free_RAM / 1.5 GiB))`. `RBITCOIN_SH_MERGE_WORKERS` overrides. +pub fn sh_extract_workers() -> usize { if let Ok(s) = std::env::var("RBITCOIN_SH_MERGE_WORKERS") { if let Ok(n) = s.parse::() { return n.clamp(1, 256); @@ -200,112 +188,79 @@ pub fn unsorted_pack_workers() -> usize { crate::sorted_run::workers_for_free_ram( crate::sorted_run::logical_cpus(), crate::sorted_run::host_mem_available_bytes().unwrap_or(0), - SH_UNSORTED_PACK_RAM_BYTES, + SH_EXTRACT_WORKER_RAM_BYTES, ) } -fn encode_unsorted_rec(sh: &[u8; 32], fk: Fk) -> [u8; UNSORTED_REC_LEN] { - let mut r = [0u8; UNSORTED_REC_LEN]; - r[..SH_HEAD_KEY_LEN].copy_from_slice(&sh[..SH_HEAD_KEY_LEN]); - r[SH_HEAD_KEY_LEN..].copy_from_slice(&fk.0.to_le_bytes()); - r +pub fn unsorted_pack_workers() -> usize { + sh_extract_workers() } -fn pwrite_all(file: &File, path: &Path, offset: u64, bytes: &[u8]) -> Result<(), StoreError> { - if bytes.is_empty() { - return Ok(()); - } - let handle = IoHandle::from_file(file); - let mut done = 0usize; - while done < bytes.len() { - let rc = handle.pwrite(offset + done as u64, &bytes[done..]); - if rc < 0 { - return Err(StoreError::io(path, io::Error::from_raw_os_error(-rc))); - } - if rc == 0 { - return Err(StoreError::io( - path, - io::Error::new(io::ErrorKind::WriteZero, "pwrite returned 0"), - )); - } - done += rc as usize; - } - Ok(()) +pub fn clear_unsorted_shard_dir(dir: &Path) { + let _ = fs::remove_dir_all(dir); } -struct UnsortedShardSink { - path: PathBuf, - file: File, - write: Mutex, +#[derive(Debug, Clone)] +pub struct UnsortedCollect { + pub recs: u64, + pub last_fk: u64, + pub per_shard: Vec, } -struct UnsortedShardWrite { - cursor: u64, - allocated: u64, +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum UnsortedCollectAction { + Full, + Append { first: u64, last: u64 }, + Skip, } -fn ensure_unsorted_allocated( - file: &File, - path: &Path, - allocated: &mut u64, - need: u64, - write_len: usize, -) -> Result<(), StoreError> { - if need <= *allocated { - return Ok(()); - } - if write_len < UNSORTED_FLUSH_BYTES { - return Ok(()); - } - let new_cap = need - .div_ceil(UNSORTED_ALLOC_STEP) - .saturating_mul(UNSORTED_ALLOC_STEP) - .max(need); - #[cfg(target_os = "linux")] - { - use std::os::unix::io::AsRawFd; - let rc = unsafe { libc::fallocate(file.as_raw_fd(), 0, 0, new_cap as i64) }; - if rc == 0 { - *allocated = new_cap; - return Ok(()); - } +fn plan_unsorted_collect( + done_last: Option, + class_a_last: u64, + any_sealed_shards: bool, + keys_present: bool, +) -> UnsortedCollectAction { + match done_last { + None => UnsortedCollectAction::Full, + Some(d) if d >= class_a_last => UnsortedCollectAction::Skip, + Some(_) if any_sealed_shards => UnsortedCollectAction::Skip, + Some(_) if !keys_present => UnsortedCollectAction::Full, + Some(d) => UnsortedCollectAction::Append { + first: d.saturating_add(1).max(1), + last: class_a_last, + }, } - file.set_len(new_cap).map_err(|e| StoreError::io(path, e))?; - *allocated = new_cap; - Ok(()) } -fn flush_unsorted_buf(sink: &UnsortedShardSink, buf: &mut Vec) -> Result<(), StoreError> { - if buf.is_empty() { - return Ok(()); - } - if !buf.len().is_multiple_of(UNSORTED_REC_LEN) { - return Err(StoreError::Corrupt( - "scripthash unsorted shard buffer not a multiple of 24", - )); - } - let n = buf.len() as u64; - let mut g = sink.write.lock().unwrap(); - let off = g.cursor; - g.cursor = g.cursor.saturating_add(n); - let need = g.cursor; - ensure_unsorted_allocated(&sink.file, &sink.path, &mut g.allocated, need, buf.len())?; - pwrite_all(&sink.file, &sink.path, off, buf)?; - buf.clear(); - Ok(()) +fn any_unsorted_keys(dir: &Path, n_shards: usize) -> bool { + (0..n_shards).any(|si| { + let p = unsorted_keys_path(dir, si); + p.is_file() || p.is_dir() + }) } -fn write_unsorted_done(dir: &Path, last_fk: u64, per_shard: &[u64]) -> Result<(), StoreError> { +struct PhaseDone { + last_fk: u64, + counts: Vec, +} + +fn write_phase_done( + dir: &Path, + name: &str, + magic: &[u8; 8], + last_fk: u64, + per_shard: &[u64], +) -> Result<(), StoreError> { let n = per_shard.len() as u32; let mut buf = Vec::with_capacity(20 + per_shard.len() * 8); - buf.extend_from_slice(UNSORTED_DONE_MAGIC); + buf.extend_from_slice(magic); buf.extend_from_slice(&n.to_le_bytes()); buf.extend_from_slice(&last_fk.to_le_bytes()); for c in per_shard { buf.extend_from_slice(&c.to_le_bytes()); } - let tmp = dir.join(format!("{UNSORTED_DONE_NAME}.tmp")); - let dst = dir.join(UNSORTED_DONE_NAME); + let tmp = dir.join(format!("{name}.tmp")); + let dst = dir.join(name); fs::write(&tmp, &buf).map_err(|e| StoreError::io(&tmp, e))?; { let f = OpenOptions::new() @@ -318,15 +273,9 @@ fn write_unsorted_done(dir: &Path, last_fk: u64, per_shard: &[u64]) -> Result<() Ok(()) } -struct UnsortedDone { - last_fk: u64, - per_shard: Vec, -} - -fn read_unsorted_done(dir: &Path, n_shards: usize) -> Option { - let p = dir.join(UNSORTED_DONE_NAME); - let buf = fs::read(&p).ok()?; - if buf.len() != 20 + n_shards * 8 || &buf[..8] != UNSORTED_DONE_MAGIC { +fn read_phase_done(dir: &Path, name: &str, magic: &[u8; 8], n_shards: usize) -> Option { + let buf = fs::read(dir.join(name)).ok()?; + if buf.len() < 20 || &buf[0..8] != magic { return None; } let n = u32::from_le_bytes(buf[8..12].try_into().ok()?) as usize; @@ -334,402 +283,1563 @@ fn read_unsorted_done(dir: &Path, n_shards: usize) -> Option { return None; } let last_fk = u64::from_le_bytes(buf[12..20].try_into().ok()?); - let mut per_shard = Vec::with_capacity(n_shards); - for i in 0..n_shards { - let off = 20 + i * 8; - let count = u64::from_le_bytes(buf[off..off + 8].try_into().ok()?); - let path = unsorted_shard_path(dir, i); - let meta = fs::metadata(&path).ok()?; - if meta.len() != count.saturating_mul(UNSORTED_REC_LEN as u64) { - return None; - } - per_shard.push(count); + if buf.len() != 20 + n * 8 { + return None; + } + let mut counts = Vec::with_capacity(n); + for i in 0..n { + let o = 20 + i * 8; + counts.push(u64::from_le_bytes(buf[o..o + 8].try_into().ok()?)); } - Some(UnsortedDone { last_fk, per_shard }) + Some(PhaseDone { last_fk, counts }) } -/// Inclusive Class A create_fk recorded in `DONE`, if the manifest is valid. pub fn unsorted_done_last_fk(dir: &Path, n_shards: usize) -> Option { - read_unsorted_done(dir, n_shards).map(|d| d.last_fk) -} - -pub fn clear_unsorted_shard_dir(dir: &Path) { - let _ = fs::remove_dir_all(dir); -} - -/// One Class A pass: n workers scan fk chunks and pwrite 24-byte recs to `n_shards` FDs. -pub fn collect_unsorted_shard_files( - store: &Store, - dir: &Path, - n_shards: usize, - workers: usize, - cancel: Option<&AtomicBool>, -) -> Result { - let last = store.txs.count(); - collect_unsorted_from_txs(&store.txs, dir, n_shards, 1, last, workers, false, cancel) + read_phase_done(dir, KEYS_DONE_NAME, KEYS_DONE_MAGIC, n_shards).map(|d| d.last_fk) } -/// Resume collect: append when `DONE` lags and no shards are sealed; skip when sealed. -pub(crate) fn collect_unsorted_covering_class_a( - store: &Store, - dir: &Path, - n_shards: usize, - workers: usize, - any_sealed_shards: bool, - cancel: Option<&AtomicBool>, -) -> Result { - let last = store.txs.count(); - let done = read_unsorted_done(dir, n_shards); - let action = plan_unsorted_collect(done.as_ref().map(|d| d.last_fk), last, any_sealed_shards); - if !matches!(action, UnsortedCollectAction::Skip) - && cancel.map(|c| c.load(Ordering::Relaxed)).unwrap_or(false) - { - return Err(StoreError::Cancelled("scripthash unsorted class a collect")); +fn leftover_legacy_unsorted(dir: &Path) -> bool { + if !dir.is_dir() { + return false; } - match action { - UnsortedCollectAction::Skip => { - let d = done.expect("skip requires a valid DONE"); - Ok(UnsortedShardCollect { - recs: d.per_shard.iter().copied().sum(), - last_fk: d.last_fk, - per_shard: d.per_shard, - }) + if dir.join(LEGACY_DONE_NAME).is_file() { + return true; + } + for i in 0..256 { + if unsorted_shard_path(dir, i).is_file() { + return true; } - UnsortedCollectAction::Full => { - collect_unsorted_from_txs(&store.txs, dir, n_shards, 1, last, workers, false, cancel) + if unsorted_keys_path(dir, i).is_file() { + return true; + } + if unsorted_post_path(dir, i).is_file() { + return true; } - UnsortedCollectAction::Append { first, last } => collect_unsorted_from_txs( - &store.txs, dir, n_shards, first, last, workers, true, cancel, - ), } + false } -#[allow(clippy::too_many_arguments)] // IO/session args stay unbundled -fn collect_unsorted_from_txs( - txs: &TxTable, - dir: &Path, - n_shards: usize, - first: u64, - last: u64, - workers: usize, - append: bool, - cancel: Option<&AtomicBool>, -) -> Result { - let n_shards = n_shards.max(1); - if !append { - clear_unsorted_shard_dir(dir); +fn chunk_ranges(first: u64, last: u64, chunk: u64) -> Vec<(u64, u64)> { + if last < first { + return Vec::new(); } - fs::create_dir_all(dir).map_err(|e| StoreError::io(dir, e))?; - let mut sinks = Vec::with_capacity(n_shards); - for shard in 0..n_shards { - let path = unsorted_shard_path(dir, shard); - let mut opts = OpenOptions::new(); - opts.create(true).write(true); - if !append { - opts.truncate(true); + let mut out = Vec::new(); + let mut cur = first; + while cur <= last { + let hi = cur.saturating_add(chunk.saturating_sub(1)).min(last); + out.push((cur, hi)); + if hi == last { + break; } - let file = opts.open(&path).map_err(|e| StoreError::io(&path, e))?; - let cursor = if append { - let len = file.metadata().map_err(|e| StoreError::io(&path, e))?.len(); - if !len.is_multiple_of(UNSORTED_REC_LEN as u64) { - return Err(StoreError::Corrupt( - "scripthash unsorted shard size not a multiple of 24", - )); - } - len + cur = hi.saturating_add(1); + } + out +} + +/// Contiguous, covering, disjoint create-fk spans. Remainder on the last span. +/// `n_workers` clamps to `[1, span]`. +fn worker_fk_spans(first: u64, last: u64, n_workers: usize) -> Vec<(u64, u64)> { + if last < first { + return Vec::new(); + } + let span = last.saturating_sub(first).saturating_add(1); + let n = (n_workers.max(1) as u64).min(span); + let base = span / n; + let rem = span % n; + let mut out = Vec::with_capacity(n as usize); + let mut cur = first; + for i in 0..n { + let len = if i + 1 == n { + base.saturating_add(rem) } else { - 0 + base }; - sinks.push(UnsortedShardSink { - path, - file, - write: Mutex::new(UnsortedShardWrite { - cursor, - allocated: cursor, - }), - }); + let hi = cur.saturating_add(len.saturating_sub(1)); + out.push((cur, hi)); + cur = hi.saturating_add(1); } + out +} - if last < first { - let per_shard: Vec = sinks - .iter() - .map(|s| s.write.lock().unwrap().cursor / UNSORTED_REC_LEN as u64) - .collect(); - write_unsorted_done(dir, last, &per_shard)?; - return Ok(UnsortedShardCollect { - recs: per_shard.iter().copied().sum(), - per_shard, - last_fk: last, - }); +/// Finished create-fks, not the max fk. A late span must not look like the scan is done. +fn note_scanned_fks(scanned: &AtomicU64, lo: u64, hi: u64) -> u64 { + let n = hi.saturating_sub(lo).saturating_add(1); + scanned.fetch_add(n, Ordering::Relaxed).saturating_add(n) +} + +fn key16_halves_bytes(bytes: &[u8]) -> (u64, u64) { + let a = u64::from_le_bytes(bytes[0..8].try_into().unwrap()); + let b = u64::from_le_bytes(bytes[8..16].try_into().unwrap()); + (a, b) +} + +/// Identity hasher for uniform key16: `finish()` is the xor of the two LE u64 halves. +/// Does not FNV-mix bytes ([`crate::int_map::U64IdentityHasher::write`]). +#[derive(Default, Clone, Copy)] +struct Key16IdentityHasher(u64); + +impl Hasher for Key16IdentityHasher { + #[inline] + fn write(&mut self, bytes: &[u8]) { + if bytes.len() == KEY16_LEN { + let (a, b) = key16_halves_bytes(bytes); + self.0 = a ^ b; + } } - let work_span = last.saturating_sub(first.saturating_sub(1)); - let workers = workers.clamp(1, 256); - let by_size = work_span.div_ceil(CLASS_A_CHUNK_FKS).max(1) as usize; - let n_chunks = by_size.max(workers.min(work_span as usize).max(1)); - let chunk_span = work_span.div_ceil(n_chunks as u64).max(1); - let workers = workers.min(n_chunks); - let next_chunk = AtomicUsize::new(0); - let stop = AtomicBool::new(false); - let first_err: Mutex> = Mutex::new(None); - let n_recs = AtomicU64::new(0); - let t0 = Instant::now(); + #[inline] + fn write_u64(&mut self, i: u64) { + if i > KEY16_LEN as u64 { + self.0 = i; + } + } - rbitcoin_log::info!( - "store: scripthash unsorted collect start n_shards={n_shards} workers={workers} \ - chunks={n_chunks} first={first} last={last} append={append}" - ); + #[inline] + fn finish(&self) -> u64 { + self.0 + } +} - std::thread::scope(|scope| { - let sinks = &sinks; - let next_chunk = &next_chunk; - let stop = &stop; - let first_err = &first_err; - let n_recs = &n_recs; - scope.spawn(|| { - let mut last_log = Instant::now(); - loop { - if stop.load(Ordering::Relaxed) { - break; - } - std::thread::sleep(Duration::from_millis(50)); - if last_log.elapsed() < MATERIALIZE_STATUS_INTERVAL { - continue; - } - last_log = Instant::now(); - let recs = n_recs.load(Ordering::Relaxed); - rbitcoin_log::info!( - "store: scripthash unsorted collect status recs≈{recs} assigned={} \ - workers={workers} elapsed={:?}", - next_chunk.load(Ordering::Relaxed).min(n_chunks), - t0.elapsed() - ); - } - }); - let mut joins = Vec::with_capacity(workers); - for _ in 0..workers { - joins.push(scope.spawn(|| { - let mut bufs: Vec> = (0..n_shards).map(|_| Vec::new()).collect(); - loop { - if stop.load(Ordering::Relaxed) - || cancel.map(|c| c.load(Ordering::Relaxed)).unwrap_or(false) - { - stop.store(true, Ordering::Relaxed); - break; - } - if first_err.lock().unwrap().is_some() { - break; - } - let i = next_chunk.fetch_add(1, Ordering::Relaxed); - if i >= n_chunks { - break; - } - let lo = first.saturating_add((i as u64).saturating_mul(chunk_span)); - let hi = lo.saturating_add(chunk_span).saturating_sub(1).min(last); - if lo > last { - continue; - } - match txs.for_each_script_hashes_in_fk_span(lo, hi, |fk, sh| { - let shard = prefix_shard_of(&sh, n_shards); - let rec = encode_unsorted_rec(&sh, fk); - bufs[shard].extend_from_slice(&rec); - n_recs.fetch_add(1, Ordering::Relaxed); - if bufs[shard].len() >= UNSORTED_FLUSH_BYTES { - flush_unsorted_buf(&sinks[shard], &mut bufs[shard])?; - } - Ok(()) - }) { - Ok(()) => {} - Err(StoreError::Cancelled(_)) => { - stop.store(true, Ordering::Relaxed); - break; - } - Err(e) => { - *first_err.lock().unwrap() = Some(e); - stop.store(true, Ordering::Relaxed); - break; - } - } - } - for (shard, buf) in bufs.iter_mut().enumerate() { - if let Err(e) = flush_unsorted_buf(&sinks[shard], buf) { - *first_err.lock().unwrap() = Some(e); - stop.store(true, Ordering::Relaxed); - } - } - })); +type Key16PackMap = HashMap>; +type KeysSpillJob = (usize, Key16PackMap); +type KeysSpillTx = SyncSender; +type KeysSpillRx = Receiver; + +fn keys_map_estimated_bytes(map: &Key16PackMap) -> u64 { + (map.len() as u64).saturating_mul(KEYS_COLLECT_BYTES_PER_KEY) +} + +fn keys_maps_estimated_bytes(maps: &[Key16PackMap]) -> u64 { + maps.iter().map(keys_map_estimated_bytes).sum() +} + +fn largest_keys_shard(maps: &[Key16PackMap]) -> Option { + maps.iter() + .enumerate() + .filter(|(_, m)| !m.is_empty()) + .max_by_key(|(_, m)| m.len()) + .map(|(i, _)| i) +} + +fn refuse_keys_leftover() -> StoreError { + StoreError::Corrupt(INDEX_REFUSE_KEYS_LEFTOVER) +} + +/// Vacant stores `word`; occupied nonzero becomes `0` (multi) unless `word` matches. +fn insert_key_word(map: &mut Key16PackMap, key: ShHeadKey, word: u64) { + match map.entry(key) { + Entry::Vacant(v) => { + v.insert(word); } - for j in joins { - if j.join().is_err() { - *first_err.lock().unwrap() = Some(StoreError::Corrupt( - "scripthash unsorted collect worker panicked", - )); + Entry::Occupied(mut o) => { + if *o.get() != 0 && (*o.get() != word || word == 0) { + o.insert(0); } } - stop.store(true, Ordering::Relaxed); - }); + } +} - if let Some(e) = first_err.lock().unwrap().take() { - return Err(e); +struct CollectTally { + recs: u64, + hits: u64, + per: Vec, +} + +impl CollectTally { + fn new(n_shards: usize) -> Self { + Self { + recs: 0, + hits: 0, + per: vec![0; n_shards], + } } - if cancel.map(|c| c.load(Ordering::Relaxed)).unwrap_or(false) { - return Err(StoreError::Cancelled("scripthash unsorted class a collect")); +} + +/// Add a worker's batch counts once. A zero tally does not touch the atomics. +fn flush_collect_tally( + tally: &mut CollectTally, + recs: &AtomicU64, + hits: Option<&AtomicU64>, + per: Option<&[AtomicU64]>, +) { + if tally.recs != 0 { + recs.fetch_add(tally.recs, Ordering::Relaxed); + tally.recs = 0; + } + if tally.hits != 0 { + if let Some(hits) = hits { + hits.fetch_add(tally.hits, Ordering::Relaxed); + } + tally.hits = 0; + } + if let Some(per) = per { + for (slot, n) in per.iter().zip(tally.per.iter_mut()) { + if *n != 0 { + slot.fetch_add(*n, Ordering::Relaxed); + *n = 0; + } + } } +} + +fn insert_key_pack(map: &mut Key16PackMap, key: ShHeadKey, fk: Fk) { + if fk.is_null() { + return; + } + insert_key_word(map, key, fk.0); +} - let mut per_shard = Vec::with_capacity(n_shards); - for sink in &sinks { - let bytes = sink.write.lock().unwrap().cursor; - if !bytes.is_multiple_of(UNSORTED_REC_LEN as u64) { - return Err(StoreError::Corrupt( - "scripthash unsorted shard size not a multiple of 24", - )); +fn encode_keys_spill(map: &Key16PackMap) -> Vec { + let mut multis = Vec::new(); + let mut singles = Vec::new(); + for (k, w) in map { + if *w == 0 { + multis.push(*k); + } else { + singles.push((*w, *k)); } - sink.file - .set_len(bytes) - .map_err(|e| StoreError::io(&sink.path, e))?; - sink.file - .sync_all() - .map_err(|e| StoreError::io(&sink.path, e))?; - per_shard.push(bytes / UNSORTED_REC_LEN as u64); - } - write_unsorted_done(dir, last, &per_shard)?; - let recs = per_shard.iter().copied().sum(); - rbitcoin_log::info!( - "store: scripthash unsorted collect done recs={recs} n_shards={n_shards} last_fk={last} \ - append={append} elapsed={:?}", - t0.elapsed() + } + singles.sort_unstable_by_key(|a| a.0); + let mut bytes = Vec::with_capacity( + KEYS_SPILL_HDR + (multis.len() + singles.len()).saturating_mul(KEY16_LEN + 8), ); - Ok(UnsortedShardCollect { - recs, - per_shard, - last_fk: last, - }) + bytes.extend_from_slice(KEYS_SPILL_MAGIC); + bytes.extend_from_slice(&(multis.len() as u32).to_le_bytes()); + bytes.extend_from_slice(&(singles.len() as u32).to_le_bytes()); + for k in &multis { + bytes.extend_from_slice(k); + } + let mut prev = 0u64; + for &(fk, k) in &singles { + write_uleb128(&mut bytes, fk.saturating_sub(prev)); + bytes.extend_from_slice(&k); + prev = fk; + } + bytes } -fn rec_fk_le(rec: &[u8; UNSORTED_REC_LEN]) -> u64 { - u64::from_le_bytes(rec[SH_HEAD_KEY_LEN..].try_into().unwrap()) +fn fold_keys_spill_bytes(map: &mut Key16PackMap, bytes: &[u8]) -> Result<(), StoreError> { + if bytes.len() < KEYS_SPILL_HDR || bytes[0..8] != KEYS_SPILL_MAGIC[..] { + return Err(refuse_keys_leftover()); + } + let n_multi = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize; + let n_single = u32::from_le_bytes(bytes[12..16].try_into().unwrap()) as usize; + let mut i = KEYS_SPILL_HDR; + for _ in 0..n_multi { + if bytes.len().saturating_sub(i) < KEY16_LEN { + return Err(StoreError::Corrupt("scripthash keys spill truncated")); + } + let mut k = [0u8; KEY16_LEN]; + k.copy_from_slice(&bytes[i..i + KEY16_LEN]); + i += KEY16_LEN; + insert_key_word(map, k, 0); + } + let mut prev = 0u64; + for _ in 0..n_single { + let (delta, n) = read_uleb128(&bytes[i..]) + .map_err(|_| StoreError::Corrupt("scripthash keys spill uleb"))?; + i = i.saturating_add(n); + if bytes.len().saturating_sub(i) < KEY16_LEN { + return Err(StoreError::Corrupt("scripthash keys spill truncated")); + } + let mut k = [0u8; KEY16_LEN]; + k.copy_from_slice(&bytes[i..i + KEY16_LEN]); + i += KEY16_LEN; + let fk = prev.saturating_add(delta); + insert_key_word(map, k, fk); + prev = fk; + } + if i != bytes.len() { + return Err(StoreError::Corrupt("scripthash keys spill trailing bytes")); + } + Ok(()) } -fn sort_unique_unsorted_recs(bytes: &mut Vec) -> Result<(), StoreError> { - if bytes.is_empty() { - return Ok(()); - } - if !bytes.len().is_multiple_of(UNSORTED_REC_LEN) { - return Err(StoreError::Corrupt( - "scripthash unsorted shard file length not a multiple of 24", - )); +fn parse_spill_seq(name: &str) -> Option { + if name.len() == 6 && name.bytes().all(|b| b.is_ascii_digit()) { + name.parse().ok() + } else { + None } - let keep = { - let (recs, rem) = bytes.as_chunks_mut::(); - debug_assert!(rem.is_empty()); - recs.sort_unstable_by(|a, b| { - a[..SH_HEAD_KEY_LEN] - .cmp(&b[..SH_HEAD_KEY_LEN]) - .then(rec_fk_le(a).cmp(&rec_fk_le(b))) +} + +fn list_keys_spill_paths(dir: &Path, shard: usize) -> Result, StoreError> { + let p = unsorted_keys_path(dir, shard); + if p.is_file() { + return Err(refuse_keys_leftover()); + } + if !p.is_dir() { + return Ok(Vec::new()); + } + let rd = fs::read_dir(&p).map_err(|e| StoreError::io(&p, e))?; + let mut seqs = Vec::new(); + for ent in rd { + let ent = ent.map_err(|e| StoreError::io(&p, e))?; + let name = ent.file_name(); + let s = name.to_string_lossy(); + if s.ends_with(".tmp") { + continue; + } + let Some(seq) = parse_spill_seq(&s) else { + return Err(refuse_keys_leftover()); + }; + seqs.push((seq, ent.path())); + } + seqs.sort_unstable_by_key(|a| a.0); + Ok(seqs.into_iter().map(|(_, path)| path).collect()) +} + +fn load_keys_shard_map(dir: &Path, shard: usize) -> Result { + let mut map = Key16PackMap::default(); + for path in list_keys_spill_paths(dir, shard)? { + let bytes = fs::read(&path).map_err(|e| StoreError::io(&path, e))?; + fold_keys_spill_bytes(&mut map, &bytes)?; + } + Ok(map) +} + +fn write_keys_spill( + dir: &Path, + shard: usize, + seq: u32, + map: &Key16PackMap, +) -> Result<(), StoreError> { + let path = unsorted_keys_spill_path(dir, shard, seq); + crate::file::write_tmp_rename(&path, &encode_keys_spill(map)) +} + +fn next_keys_spill_seq(dir: &Path, shard: usize) -> Result { + let paths = list_keys_spill_paths(dir, shard)?; + let mut max_seq = None; + for p in &paths { + let name = p.file_name().and_then(|n| n.to_str()).unwrap_or(""); + if let Some(seq) = parse_spill_seq(name) { + max_seq = Some(max_seq.map_or(seq, |m: u32| m.max(seq))); + } + } + Ok(max_seq.map(|s| s.saturating_add(1)).unwrap_or(0)) +} + +#[cfg(test)] +pub(crate) fn write_keys_spill_entries( + dir: &Path, + shard: usize, + seq: u32, + items: &[(ShHeadKey, Option)], +) -> Result<(), StoreError> { + let mut map = Key16PackMap::default(); + for &(k, first) in items { + match first { + Some(fk) => insert_key_pack(&mut map, k, fk), + None => insert_key_word(&mut map, k, 0), + } + } + write_keys_spill(dir, shard, seq, &map) +} + +fn unlink_keys_shard(dir: &Path, shard: usize) { + let p = unsorted_keys_path(dir, shard); + if p.is_dir() { + let _ = fs::remove_dir_all(&p); + } else if p.is_file() { + let _ = fs::remove_file(&p); + } +} + +type Key16FkMap = HashMap, BuildHasherDefault>; + +#[derive(Default, Debug)] +struct PostPackMap { + map: Key16FkMap, + n_fks: usize, +} + +impl PostPackMap { + fn estimated_bytes(&self) -> usize { + self.map + .len() + .saturating_mul(POST_MAP_KEY_BYTES) + .saturating_add(self.n_fks.saturating_mul(POST_MAP_FK_BYTES)) + } + + fn is_empty(&self) -> bool { + self.map.is_empty() + } +} + +fn refuse_posts_leftover() -> StoreError { + StoreError::Corrupt(INDEX_REFUSE_POST_LEFTOVER) +} + +fn posts_maps_estimated_bytes(maps: &[PostPackMap]) -> u64 { + maps.iter().map(|m| m.estimated_bytes() as u64).sum() +} + +fn largest_posts_shard(maps: &[PostPackMap]) -> Option { + maps.iter() + .enumerate() + .filter(|(_, m)| !m.is_empty()) + .max_by_key(|(_, m)| m.estimated_bytes()) + .map(|(i, _)| i) +} + +fn fks_strictly_increasing(fks: &[u64]) -> bool { + fks.windows(2).all(|w| w[0] < w[1]) +} + +/// Keep `Vec` strictly increasing. Scan order appends; an older fk inserts. +fn insert_post_fk(map: &mut PostPackMap, key: ShHeadKey, fk: u64) { + if fk == 0 { + return; + } + match map.map.entry(key) { + Entry::Vacant(v) => { + v.insert(vec![fk]); + map.n_fks = map.n_fks.saturating_add(1); + } + Entry::Occupied(mut o) => { + let v = o.get_mut(); + if let Some(last) = v.last().copied() { + if fk > last { + v.push(fk); + map.n_fks = map.n_fks.saturating_add(1); + return; + } + if fk == last { + return; + } + } + match v.binary_search(&fk) { + Ok(_) => {} + Err(i) => { + v.insert(i, fk); + map.n_fks = map.n_fks.saturating_add(1); + } + } + } + } +} + +/// Linear merge of two strictly increasing fk runs. Equals collapse. +fn merge_sorted_unique_fks(dst: &mut Vec, src: Vec) { + if src.is_empty() { + return; + } + if dst.is_empty() { + *dst = src; + dst.dedup(); + return; + } + debug_assert!(fks_strictly_increasing(dst)); + debug_assert!(fks_strictly_increasing(&src)); + let mut out = Vec::with_capacity(dst.len().saturating_add(src.len())); + let mut i = 0usize; + let mut j = 0usize; + while i < dst.len() || j < src.len() { + let take = match (dst.get(i), src.get(j)) { + (Some(&a), Some(&b)) if a < b => { + i += 1; + a + } + (Some(&a), Some(&b)) if a > b => { + j += 1; + b + } + (Some(&a), Some(_)) => { + i += 1; + j += 1; + a + } + (Some(&a), None) => { + i += 1; + a + } + (None, Some(&b)) => { + j += 1; + b + } + (None, None) => break, + }; + if out.last().copied() != Some(take) { + out.push(take); + } + } + *dst = out; +} + +fn fold_post_map(dst: &mut PostPackMap, src: PostPackMap) { + for (k, fks) in src.map { + match dst.map.entry(k) { + Entry::Vacant(v) => { + let mut f = fks; + f.dedup(); + debug_assert!(fks_strictly_increasing(&f)); + dst.n_fks = dst.n_fks.saturating_add(f.len()); + v.insert(f); + } + Entry::Occupied(mut o) => { + let before = o.get().len(); + merge_sorted_unique_fks(o.get_mut(), fks); + dst.n_fks = dst + .n_fks + .saturating_sub(before) + .saturating_add(o.get().len()); + } + } + } +} + +fn encode_posts_spill(map: &PostPackMap) -> Vec { + let mut bytes = Vec::with_capacity( + POSTS_SPILL_HDR + + map + .map + .len() + .saturating_mul(KEY16_LEN + 8) + .saturating_add(map.n_fks.saturating_mul(8)), + ); + bytes.extend_from_slice(POSTS_SPILL_MAGIC); + bytes.extend_from_slice(&(map.map.len() as u32).to_le_bytes()); + for (k, fks) in &map.map { + debug_assert!(fks_strictly_increasing(fks)); + bytes.extend_from_slice(k); + write_uleb128(&mut bytes, fks.len() as u64); + let mut prev = 0u64; + for &fk in fks { + write_uleb128(&mut bytes, fk.saturating_sub(prev)); + prev = fk; + } + } + bytes +} + +fn fold_posts_spill_bytes(map: &mut PostPackMap, bytes: &[u8]) -> Result<(), StoreError> { + if bytes.len() < POSTS_SPILL_HDR || bytes[0..8] != POSTS_SPILL_MAGIC[..] { + return Err(refuse_posts_leftover()); + } + let n_keys = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize; + let mut i = POSTS_SPILL_HDR; + let mut src = PostPackMap::default(); + for _ in 0..n_keys { + if bytes.len().saturating_sub(i) < KEY16_LEN { + return Err(StoreError::Corrupt("scripthash posts spill truncated")); + } + let mut k = [0u8; KEY16_LEN]; + k.copy_from_slice(&bytes[i..i + KEY16_LEN]); + i += KEY16_LEN; + let (n, nread) = read_uleb128(&bytes[i..]) + .map_err(|_| StoreError::Corrupt("scripthash posts spill uleb"))?; + i = i.saturating_add(nread); + let mut fks = Vec::with_capacity(n as usize); + let mut prev = 0u64; + for _ in 0..n { + let (delta, nread) = read_uleb128(&bytes[i..]) + .map_err(|_| StoreError::Corrupt("scripthash posts spill uleb"))?; + i = i.saturating_add(nread); + let fk = prev.saturating_add(delta); + fks.push(fk); + prev = fk; + } + src.map.insert(k, fks); + src.n_fks = src.n_fks.saturating_add(n as usize); + } + if i != bytes.len() { + return Err(StoreError::Corrupt("scripthash posts spill trailing bytes")); + } + fold_post_map(map, src); + Ok(()) +} + +fn list_posts_spill_paths(dir: &Path, shard: usize) -> Result, StoreError> { + let p = unsorted_post_path(dir, shard); + if p.is_file() { + return Err(refuse_posts_leftover()); + } + if !p.is_dir() { + return Ok(Vec::new()); + } + let rd = fs::read_dir(&p).map_err(|e| StoreError::io(&p, e))?; + let mut seqs = Vec::new(); + for ent in rd { + let ent = ent.map_err(|e| StoreError::io(&p, e))?; + let name = ent.file_name(); + let s = name.to_string_lossy(); + if s.ends_with(".tmp") { + continue; + } + let Some(seq) = parse_spill_seq(&s) else { + return Err(refuse_posts_leftover()); + }; + seqs.push((seq, ent.path())); + } + seqs.sort_unstable_by_key(|a| a.0); + Ok(seqs.into_iter().map(|(_, path)| path).collect()) +} + +fn load_post_shard_map(dir: &Path, shard: usize) -> Result { + let mut map = PostPackMap::default(); + for path in list_posts_spill_paths(dir, shard)? { + let bytes = fs::read(&path).map_err(|e| StoreError::io(&path, e))?; + fold_posts_spill_bytes(&mut map, &bytes)?; + } + Ok(map) +} + +fn write_posts_spill( + dir: &Path, + shard: usize, + seq: u32, + map: &PostPackMap, +) -> Result<(), StoreError> { + let path = unsorted_post_spill_path(dir, shard, seq); + crate::file::write_tmp_rename(&path, &encode_posts_spill(map)) +} + +fn next_post_spill_seq(dir: &Path, shard: usize) -> Result { + let paths = list_posts_spill_paths(dir, shard)?; + let mut max_seq = None; + for p in &paths { + let name = p.file_name().and_then(|n| n.to_str()).unwrap_or(""); + if let Some(seq) = parse_spill_seq(name) { + max_seq = Some(max_seq.map_or(seq, |m: u32| m.max(seq))); + } + } + Ok(max_seq.map(|s| s.saturating_add(1)).unwrap_or(0)) +} + +fn spill_post_shard_map( + dir: &Path, + shard: usize, + seqs: &[AtomicU32], + map: PostPackMap, +) -> Result<(), StoreError> { + if map.is_empty() { + return Ok(()); + } + let seq = seqs[shard].fetch_add(1, Ordering::Relaxed); + write_posts_spill(dir, shard, seq, &map) +} + +fn spill_largest_posts_while_over( + maps: &mut [PostPackMap], + budget: u64, + mut spill: impl FnMut(usize, PostPackMap) -> Result<(), StoreError>, +) -> Result<(), StoreError> { + while posts_maps_estimated_bytes(maps) >= budget { + let Some(si) = largest_posts_shard(maps) else { + break; + }; + let taken = std::mem::take(&mut maps[si]); + spill(si, taken)?; + } + Ok(()) +} + +fn spill_all_post_maps( + maps: &mut [PostPackMap], + mut spill: impl FnMut(usize, PostPackMap) -> Result<(), StoreError>, +) -> Result<(), StoreError> { + for (si, local) in maps.iter_mut().enumerate() { + let taken = std::mem::take(local); + spill(si, taken)?; + } + Ok(()) +} + +type PostSpillJob = (usize, PostPackMap); +type PostSpillTx = SyncSender; +type PostSpillRx = Receiver; + +fn posts_spill_channel() -> (PostSpillTx, PostSpillRx) { + mpsc::sync_channel(1) +} + +fn send_posts_spill(tx: &PostSpillTx, shard: usize, map: PostPackMap) -> Result<(), StoreError> { + if map.is_empty() { + return Ok(()); + } + tx.send((shard, map)) + .map_err(|_| StoreError::Corrupt("scripthash post spill writer stopped")) +} + +fn posts_spill_writer( + rx: PostSpillRx, + dir: &Path, + seqs: &[AtomicU32], + err: &Mutex>, +) { + while let Ok((si, map)) = rx.recv() { + if err.lock().unwrap().is_some() { + continue; + } + if let Err(e) = spill_post_shard_map(dir, si, seqs, map) { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(e); + } + } + } +} + +fn unlink_post_shard(dir: &Path, shard: usize) { + let p = unsorted_post_path(dir, shard); + if p.is_dir() { + let _ = fs::remove_dir_all(&p); + } else if p.is_file() { + let _ = fs::remove_file(&p); + } +} + +#[cfg(test)] +pub(crate) fn write_post_spill_entries( + dir: &Path, + shard: usize, + seq: u32, + items: &[(ShHeadKey, &[u64])], +) -> Result<(), StoreError> { + let mut map = PostPackMap::default(); + for &(k, fks) in items { + for &fk in fks { + insert_post_fk(&mut map, k, fk); + } + } + write_posts_spill(dir, shard, seq, &map) +} + +#[cfg(test)] +pub(crate) fn load_post_shard_entries( + dir: &Path, + shard: usize, +) -> Result)>, StoreError> { + let map = load_post_shard_map(dir, shard)?; + Ok(map.map.into_iter().collect()) +} + +fn finish_key_shard( + table: &ScriptHashTable, + dir: &Path, + shard: usize, + n_shards: usize, +) -> Result<(u64, u64, u64), StoreError> { + let map = load_keys_shard_map(dir, shard)?; + let times = seal_pack_map_to_head_and_fuse(table, dir, shard, n_shards, map)?; + unlink_keys_shard(dir, shard); + Ok(times) +} + +fn spill_shard_map( + dir: &Path, + shard: usize, + seqs: &[AtomicU32], + map: Key16PackMap, +) -> Result<(), StoreError> { + if map.is_empty() { + return Ok(()); + } + let seq = seqs[shard].fetch_add(1, Ordering::Relaxed); + write_keys_spill(dir, shard, seq, &map) +} + +fn spill_largest_keys_while_over( + maps: &mut [Key16PackMap], + budget: u64, + mut spill: impl FnMut(usize, Key16PackMap) -> Result<(), StoreError>, +) -> Result<(), StoreError> { + while keys_maps_estimated_bytes(maps) >= budget { + let Some(si) = largest_keys_shard(maps) else { + break; + }; + let taken = std::mem::take(&mut maps[si]); + spill(si, taken)?; + } + Ok(()) +} + +fn spill_all_keys_maps( + maps: &mut [Key16PackMap], + mut spill: impl FnMut(usize, Key16PackMap) -> Result<(), StoreError>, +) -> Result<(), StoreError> { + for (si, local) in maps.iter_mut().enumerate() { + let taken = std::mem::take(local); + spill(si, taken)?; + } + Ok(()) +} + +/// One in-flight map. The next spill blocks until the writer takes this one. +fn keys_spill_channel() -> (KeysSpillTx, KeysSpillRx) { + mpsc::sync_channel(1) +} + +fn send_keys_spill(tx: &KeysSpillTx, shard: usize, map: Key16PackMap) -> Result<(), StoreError> { + if map.is_empty() { + return Ok(()); + } + tx.send((shard, map)) + .map_err(|_| StoreError::Corrupt("scripthash keys spill writer stopped")) +} + +fn keys_spill_writer( + rx: KeysSpillRx, + dir: &Path, + seqs: &[AtomicU32], + err: &Mutex>, +) { + while let Ok((si, map)) = rx.recv() { + if err.lock().unwrap().is_some() { + continue; + } + if let Err(e) = spill_shard_map(dir, si, seqs, map) { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(e); + } + } + } +} + +fn collect_keys_from_txs( + txs: &TxTable, + table: &ScriptHashTable, + dir: &Path, + first: u64, + last: u64, + workers: usize, + cancel: Option<&AtomicBool>, +) -> Result { + let n_shards = table.head_shard_count().max(1); + if last < first { + return Ok(UnsortedCollect { + recs: 0, + last_fk: last, + per_shard: vec![0; n_shards], }); - let mut w = 0usize; - for r in 0..recs.len() { - if Fk(rec_fk_le(&recs[r])).is_null() { - continue; + } + fs::create_dir_all(dir.join(KEYS_SUBDIR)).map_err(|e| StoreError::io(dir, e))?; + let budget = SH_EXTRACT_WORKER_RAM_BYTES; + let seqs: Vec = { + let mut v = Vec::with_capacity(n_shards); + for si in 0..n_shards { + v.push(AtomicU32::new(next_keys_spill_seq(dir, si)?)); + } + v + }; + let recs = AtomicU64::new(0); + let scanned = AtomicU64::new(0); + let span_fks = last.saturating_sub(first).saturating_add(1); + let err = Mutex::new(None::); + let t0 = Instant::now(); + let last_status = Mutex::new(t0); + let spans = worker_fk_spans(first, last, workers); + std::thread::scope(|scope| { + let n_workers = spans.len().max(1); + rbitcoin_log::info!( + "store: scripthash keys collect start workers={n_workers} budget_MiB={} fk={first}..{last}", + budget / (1 << 20) + ); + let (tx, rx) = keys_spill_channel(); + scope.spawn(|| keys_spill_writer(rx, dir, &seqs, &err)); + for &(span_lo, span_hi) in &spans { + let tx = tx.clone(); + let recs = &recs; + let scanned = &scanned; + let err = &err; + let last_status = &last_status; + scope.spawn(move || { + let mut locals: Vec = + (0..n_shards).map(|_| Key16PackMap::default()).collect(); + let mut tally = CollectTally::new(0); + for (lo, hi) in chunk_ranges(span_lo, span_hi, CLASS_A_CHUNK_FKS) { + if check_cancel(cancel, "scripthash keys collect").is_err() { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(StoreError::Cancelled("scripthash keys collect")); + } + break; + } + if err.lock().unwrap().is_some() { + break; + } + let r = txs.for_each_script_hashes_in_fk_span(lo, hi, |fk, sh| { + let si = prefix_shard_of(&sh, n_shards); + tally.recs = tally.recs.saturating_add(1); + insert_key_pack(&mut locals[si], head_key_from_full(&sh), fk); + Ok(()) + }); + flush_collect_tally(&mut tally, recs, None, None); + if let Err(e) = r { + *err.lock().unwrap() = Some(e); + break; + } + if let Err(e) = spill_largest_keys_while_over(&mut locals, budget, |si, map| { + send_keys_spill(&tx, si, map) + }) { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(e); + } + break; + } + let done = note_scanned_fks(scanned, lo, hi); + let now = Instant::now(); + let mut st = last_status.lock().unwrap(); + if now.duration_since(*st) >= MATERIALIZE_STATUS_INTERVAL { + *st = now; + rbitcoin_log::info!( + "store: scripthash keys collect scanned={done}/{span_fks} recs={} elapsed={:?}", + recs.load(Ordering::Relaxed), + t0.elapsed() + ); + } + } + if err.lock().unwrap().is_none() { + rbitcoin_log::info!( + "store: scripthash keys collect flush span={span_lo}..{span_hi}" + ); + if let Err(e) = spill_all_keys_maps(&mut locals, |si, map| { + send_keys_spill(&tx, si, map) + }) { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(e); + } + } + } + }); + } + drop(tx); + }); + if let Some(e) = err.lock().unwrap().take() { + return Err(e); + } + let merge_workers = workers.max(1).min(n_shards.max(1)); + fs::create_dir_all(dir.join(MULTI_SUBDIR)).map_err(|e| StoreError::io(dir, e))?; + rbitcoin_log::info!( + "store: scripthash keys merge start n_shards={n_shards} workers={merge_workers}" + ); + let t_merge = Instant::now(); + let per_shard: Vec = (0..n_shards).map(|_| AtomicU64::new(0)).collect(); + if merge_workers <= 1 { + for (si, slot) in per_shard.iter().enumerate() { + check_cancel(cancel, "scripthash keys merge")?; + let (n, fold_ns, bdz_ns) = finish_key_shard(table, dir, si, n_shards)?; + slot.store(n, Ordering::Relaxed); + rbitcoin_log::info!( + "store: scripthash keys merge shard={si:02x} keys={n} fold={:?} bdz={:?}", + Duration::from_nanos(fold_ns), + Duration::from_nanos(bdz_ns) + ); + } + } else { + let merge_err = Mutex::new(None::); + let merge_jobs = Mutex::new(VecDeque::from_iter(0..n_shards)); + std::thread::scope(|scope| { + for _ in 0..merge_workers { + let merge_jobs = &merge_jobs; + let merge_err = &merge_err; + let per_shard = &per_shard; + scope.spawn(move || loop { + if check_cancel(cancel, "scripthash keys merge").is_err() { + let mut g = merge_err.lock().unwrap(); + if g.is_none() { + *g = Some(StoreError::Cancelled("scripthash keys merge")); + } + break; + } + if merge_err.lock().unwrap().is_some() { + break; + } + let si = merge_jobs.lock().unwrap().pop_front(); + let Some(si) = si else { + break; + }; + match finish_key_shard(table, dir, si, n_shards) { + Ok((n, fold_ns, bdz_ns)) => { + per_shard[si].store(n, Ordering::Relaxed); + rbitcoin_log::info!( + "store: scripthash keys merge shard={si:02x} keys={n} fold={:?} bdz={:?}", + Duration::from_nanos(fold_ns), + Duration::from_nanos(bdz_ns) + ); + } + Err(e) => { + *merge_err.lock().unwrap() = Some(e); + break; + } + } + }); } - if w > 0 && recs[r] == recs[w - 1] { - continue; + }); + if let Some(e) = merge_err.lock().unwrap().take() { + return Err(e); + }; + } + let per_shard: Vec = per_shard + .iter() + .map(|c| c.load(Ordering::Relaxed)) + .collect(); + rbitcoin_log::info!( + "store: scripthash keys merge done elapsed={:?}", + t_merge.elapsed() + ); + write_phase_done(dir, KEYS_DONE_NAME, KEYS_DONE_MAGIC, last, &per_shard)?; + Ok(UnsortedCollect { + recs: recs.load(Ordering::Relaxed), + last_fk: last, + per_shard, + }) +} + +pub fn collect_unsorted_shard_files( + store: &Store, + dir: &Path, + n_shards: usize, + workers: usize, + cancel: Option<&AtomicBool>, +) -> Result { + collect_unsorted_covering_class_a(store, dir, n_shards, workers, false, cancel) +} + +pub(crate) fn collect_unsorted_covering_class_a( + store: &Store, + dir: &Path, + n_shards: usize, + workers: usize, + any_sealed_shards: bool, + cancel: Option<&AtomicBool>, +) -> Result { + collect_unsorted_covering_txs( + &store.txs, + &store.scripthash, + dir, + n_shards, + workers, + any_sealed_shards, + cancel, + ) +} + +pub(crate) fn collect_unsorted_covering_txs( + txs: &TxTable, + table: &ScriptHashTable, + dir: &Path, + n_shards: usize, + workers: usize, + any_sealed_shards: bool, + cancel: Option<&AtomicBool>, +) -> Result { + let last = txs.count(); + if leftover_legacy_unsorted(dir) + && read_phase_done(dir, KEYS_DONE_NAME, KEYS_DONE_MAGIC, n_shards).is_none() + { + clear_unsorted_shard_dir(dir); + } + let done = read_phase_done(dir, KEYS_DONE_NAME, KEYS_DONE_MAGIC, n_shards); + let keys_present = any_unsorted_keys(dir, n_shards); + let action = plan_unsorted_collect( + done.as_ref().map(|d| d.last_fk), + last, + any_sealed_shards, + keys_present, + ); + check_cancel(cancel, "scripthash keys collect")?; + match action { + UnsortedCollectAction::Skip => { + let d = done.expect("skip requires DONE.keys"); + Ok(UnsortedCollect { + recs: d.counts.iter().sum(), + last_fk: d.last_fk, + per_shard: d.counts, + }) + } + UnsortedCollectAction::Full => { + clear_unsorted_shard_dir(dir); + collect_keys_from_txs(txs, table, dir, 1, last, workers, cancel) + } + UnsortedCollectAction::Append { first, last } => { + collect_keys_from_txs(txs, table, dir, first, last, workers, cancel) + } + } +} + +fn write_multi_fuse_keys(dir: &Path, si: usize, dupes: &[u64]) -> Result { + let path = unsorted_multi_fuse_path(dir, si); + if dupes.is_empty() { + let _ = fs::remove_file(&path); + return Ok(0); + } + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).map_err(|e| StoreError::io(parent, e))?; + } + let fuse = SealedFuse8::build(dupes)?; + fuse.write_to(&path)?; + Ok(dupes.len() as u64) +} + +fn seal_pack_map_to_head_and_fuse( + table: &ScriptHashTable, + dir: &Path, + si: usize, + n_shards: usize, + map: Key16PackMap, +) -> Result<(u64, u64, u64), StoreError> { + let nkeys = map.len() as u64; + let t_fold = Instant::now(); + let n = map.len(); + let mut recs = Vec::with_capacity(n); + let mut mixed = Vec::with_capacity(n); + let mut dupes = Vec::new(); + let mut n_singles = 0u64; + // Consuming the map drops its buckets before fuse8 and BDZ. + for (k, w) in map { + let m = mix_key16(&k); + mixed.push(m); + if w == 0 { + dupes.push(m); + } else { + n_singles += 1; + } + recs.push((k, w)); + } + let _n_fuse = write_multi_fuse_keys(dir, si, &dupes)?; + let fold_ns = t_fold.elapsed().as_nanos() as u64; + let t_bdz = Instant::now(); + let head = sorted_main_shard_path(table.store_dir(), si, n_shards); + let _ = fs::remove_file(mphf_path(&head)); + let _ = fs::remove_file(val_path(&head)); + MphfHead::write_pack8_mixed(&head, &recs, &mixed)?; + if si < table.head_shard_count() { + table.set_extract_inline_creates(si, n_singles)?; + } + let bdz_ns = t_bdz.elapsed().as_nanos() as u64; + Ok((nkeys, fold_ns, bdz_ns)) +} + +pub(crate) fn seal_mphf_from_keys( + table: &ScriptHashTable, + dir: &Path, + n_shards: usize, + cancel: Option<&AtomicBool>, +) -> Result<(), StoreError> { + fs::create_dir_all(dir.join(MULTI_SUBDIR)).map_err(|e| StoreError::io(dir, e))?; + let unsealed: HashSet = table.unsealed_main_shards().into_iter().collect(); + let jobs: Vec = (0..n_shards) + .filter(|&si| { + if !unsealed.contains(&si) { + unlink_keys_shard(dir, si); + return false; } - if w != r { - recs[w] = recs[r]; + let p = unsorted_keys_path(dir, si); + p.is_dir() || p.is_file() + }) + .collect(); + let remaining = jobs.len(); + let t_all = Instant::now(); + let workers = sh_extract_workers().max(1).min(remaining.max(1)); + if remaining > 0 { + rbitcoin_log::info!( + "store: scripthash keys merge start remaining={remaining} n_shards={n_shards} workers={workers}" + ); + } + let err = Mutex::new(None::); + let jobq = Mutex::new(VecDeque::from(jobs)); + std::thread::scope(|scope| { + for _ in 0..workers.min(remaining.max(1)) { + if remaining == 0 { + break; + } + let jobq = &jobq; + let err = &err; + scope.spawn(move || loop { + if check_cancel(cancel, "scripthash mphf from keys").is_err() { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(StoreError::Cancelled("scripthash mphf from keys")); + } + break; + } + if err.lock().unwrap().is_some() { + break; + } + let Some(si) = jobq.lock().unwrap().pop_front() else { + break; + }; + match seal_one_mphf_shard(table, dir, si, n_shards) { + Ok((nkeys, fold_ns, bdz_ns)) => { + rbitcoin_log::info!( + "store: scripthash keys merge shard={si:02x} keys={nkeys} fold={:?} bdz={:?}", + Duration::from_nanos(fold_ns), + Duration::from_nanos(bdz_ns) + ); + } + Err(e) => { + *err.lock().unwrap() = Some(e); + break; + } + } + }); + } + }); + if let Some(e) = err.lock().unwrap().take() { + return Err(e); + } + if remaining > 0 { + rbitcoin_log::info!( + "store: scripthash keys merge done remaining={remaining} elapsed={:?}", + t_all.elapsed() + ); + } + Ok(()) +} + +fn seal_one_mphf_shard( + table: &ScriptHashTable, + dir: &Path, + si: usize, + n_shards: usize, +) -> Result<(usize, u64, u64), StoreError> { + let map = load_keys_shard_map(dir, si)?; + let (nkeys, fold_ns, bdz_ns) = seal_pack_map_to_head_and_fuse(table, dir, si, n_shards, map)?; + unlink_keys_shard(dir, si); + Ok((nkeys as usize, fold_ns, bdz_ns)) +} + +fn open_multi_fuses(dir: &Path, n_shards: usize) -> Result>, StoreError> { + let mut out = Vec::with_capacity(n_shards); + for si in 0..n_shards { + let p = unsorted_multi_fuse_path(dir, si); + if p.is_file() { + out.push(Some(SealedFuse8::read_from(&p)?)); + } else { + out.push(None); + } + } + Ok(out) +} + +fn leftover_post_file(dir: &Path) -> bool { + (0..256).any(|i| unsorted_post_path(dir, i).is_file()) +} + +fn collect_posts_from_txs( + txs: &TxTable, + table: &ScriptHashTable, + dir: &Path, + first: u64, + last: u64, + workers: usize, + cancel: Option<&AtomicBool>, +) -> Result { + let n_shards = table.head_shard_count().max(1); + if last < first { + return Ok(UnsortedCollect { + recs: 0, + last_fk: last, + per_shard: vec![0; n_shards], + }); + } + if leftover_post_file(dir) { + return Err(refuse_posts_leftover()); + } + fs::create_dir_all(dir.join(POST_SUBDIR)).map_err(|e| StoreError::io(dir, e))?; + let fuses = Arc::new(open_multi_fuses(dir, n_shards)?); + let budget = SH_EXTRACT_WORKER_RAM_BYTES; + let seqs: Vec = { + let mut v = Vec::with_capacity(n_shards); + for si in 0..n_shards { + v.push(AtomicU32::new(next_post_spill_seq(dir, si)?)); + } + v + }; + let recs = AtomicU64::new(0); + let hits = AtomicU64::new(0); + let scanned = AtomicU64::new(0); + let span_fks = last.saturating_sub(first).saturating_add(1); + let per = (0..n_shards).map(|_| AtomicU64::new(0)).collect::>(); + let err = Mutex::new(None::); + let t0 = Instant::now(); + let last_status = Mutex::new(t0); + let spans = worker_fk_spans(first, last, workers); + std::thread::scope(|scope| { + let n_workers = spans.len().max(1); + rbitcoin_log::info!( + "store: scripthash postings collect start workers={n_workers} budget_MiB={} fk={first}..{last}", + budget / (1 << 20) + ); + let (tx, rx) = posts_spill_channel(); + scope.spawn(|| posts_spill_writer(rx, dir, &seqs, &err)); + for &(span_lo, span_hi) in &spans { + let tx = tx.clone(); + let fuses = Arc::clone(&fuses); + let recs = &recs; + let hits = &hits; + let scanned = &scanned; + let per = &per; + let err = &err; + let last_status = &last_status; + scope.spawn(move || { + let mut locals: Vec = + (0..n_shards).map(|_| PostPackMap::default()).collect(); + let mut tally = CollectTally::new(n_shards); + for (lo, hi) in chunk_ranges(span_lo, span_hi, CLASS_A_CHUNK_FKS) { + if check_cancel(cancel, "scripthash postings collect").is_err() { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(StoreError::Cancelled("scripthash postings collect")); + } + break; + } + if err.lock().unwrap().is_some() { + break; + } + let r = txs.for_each_script_hashes_in_fk_span(lo, hi, |fk, sh| { + tally.recs = tally.recs.saturating_add(1); + let si = prefix_shard_of(&sh, n_shards); + let Some(fuse) = fuses[si].as_ref() else { + return Ok(()); + }; + let key = head_key_from_full(&sh); + if !fuse.contains(mix_key16(&key)) { + return Ok(()); + } + insert_post_fk(&mut locals[si], key, fk.0); + tally.hits = tally.hits.saturating_add(1); + tally.per[si] = tally.per[si].saturating_add(1); + Ok(()) + }); + flush_collect_tally(&mut tally, recs, Some(hits), Some(per)); + if let Err(e) = r { + *err.lock().unwrap() = Some(e); + break; + } + if let Err(e) = + spill_largest_posts_while_over(&mut locals, budget, |si, map| { + send_posts_spill(&tx, si, map) + }) + { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(e); + } + break; + } + let done = note_scanned_fks(scanned, lo, hi); + let now = Instant::now(); + let mut st = last_status.lock().unwrap(); + if now.duration_since(*st) >= MATERIALIZE_STATUS_INTERVAL { + *st = now; + rbitcoin_log::info!( + "store: scripthash postings collect scanned={done}/{span_fks} recs={} hits={} elapsed={:?}", + recs.load(Ordering::Relaxed), + hits.load(Ordering::Relaxed), + t0.elapsed() + ); + } + } + if err.lock().unwrap().is_none() { + rbitcoin_log::info!( + "store: scripthash postings collect flush span={span_lo}..{span_hi}" + ); + if let Err(e) = spill_all_post_maps(&mut locals, |si, map| { + send_posts_spill(&tx, si, map) + }) { + let mut g = err.lock().unwrap(); + if g.is_none() { + *g = Some(e); + } + } + } + }); + } + drop(tx); + }); + if let Some(e) = err.lock().unwrap().take() { + return Err(e); + } + let per_shard: Vec = per.iter().map(|a| a.load(Ordering::Relaxed)).collect(); + write_phase_done(dir, POST_DONE_NAME, POST_DONE_MAGIC, last, &per_shard)?; + Ok(UnsortedCollect { + recs: recs.load(Ordering::Relaxed), + last_fk: last, + per_shard, + }) +} + +pub(crate) fn collect_posts_covering( + txs: &TxTable, + table: &ScriptHashTable, + dir: &Path, + workers: usize, + any_sealed_shards: bool, + cancel: Option<&AtomicBool>, +) -> Result { + if leftover_post_file(dir) { + return Err(refuse_posts_leftover()); + } + let n_shards = table.head_shard_count().max(1); + let last = txs.count(); + let done = read_phase_done(dir, POST_DONE_NAME, POST_DONE_MAGIC, n_shards); + let action = plan_unsorted_collect( + done.as_ref().map(|d| d.last_fk), + last, + any_sealed_shards, + true, + ); + check_cancel(cancel, "scripthash postings collect")?; + match action { + UnsortedCollectAction::Skip => { + let d = done.expect("skip requires DONE.post"); + Ok(UnsortedCollect { + recs: d.counts.iter().sum(), + last_fk: d.last_fk, + per_shard: d.counts, + }) + } + UnsortedCollectAction::Full => { + let post_dir = dir.join(POST_SUBDIR); + let _ = fs::remove_dir_all(&post_dir); + collect_posts_from_txs(txs, table, dir, 1, last, workers, cancel) + } + UnsortedCollectAction::Append { first, last } => { + collect_posts_from_txs(txs, table, dir, first, last, workers, cancel) + } + } +} + +fn pack_post_shard( + table: &ScriptHashTable, + unsorted_dir: &Path, + shard: usize, + cancel: Option<&AtomicBool>, +) -> Result { + check_cancel(cancel, "scripthash unsorted shard pack")?; + let mut session = table.pack_shard_session(shard)?; + let map = load_post_shard_map(unsorted_dir, shard)?; + if map.is_empty() { + return session.finish_pack(); + } + let n_shards = table.head_shard_count().max(1); + let mphf = MphfHead::open(sorted_main_shard_path(table.store_dir(), shard, n_shards))?; + session.reserve_pack_recs(map.n_fks); + let mut fp_singles = 0u64; + for (k, fks) in map.map { + check_cancel(cancel, "scripthash unsorted shard pack")?; + debug_assert!(fks_strictly_increasing(&fks)); + if fks.len() <= 1 { + if fks.len() == 1 { + fp_singles = fp_singles.saturating_add(1); } - w += 1; + continue; } - w - }; - bytes.truncate(keep.saturating_mul(UNSORTED_REC_LEN)); + let slot = mphf.slot_for_key16(&k)?; + for fk in fks { + session.push_sorted_slot_fk(slot, Fk(fk))?; + } + } + let mut pack = session.finish_pack()?; + pack.fp_singles = fp_singles; + Ok(pack) +} + +fn seal_shard( + table: &ScriptHashTable, + shard: usize, + pack: ShShardPack, + max_fk: &AtomicU64, + progress: &MaterializeProgress, +) -> Result<(), StoreError> { + max_fk.fetch_max(pack.max_fk, Ordering::Relaxed); + let creates = pack.creates; + let t_mphf = Instant::now(); + table.publish_packed_shard(shard, pack)?; + progress + .mphf_ns + .fetch_add(t_mphf.elapsed().as_nanos() as u64, Ordering::Relaxed); + progress.shards_published.fetch_add(1, Ordering::Relaxed); + progress + .creates_published + .fetch_add(creates, Ordering::Relaxed); + table.store_sharded_cold_progress( + progress.keys_packed.load(Ordering::Relaxed), + progress.creates_published.load(Ordering::Relaxed), + )?; Ok(()) } -fn unique_sh_key_count(bytes: &[u8]) -> usize { - let mut n = 0usize; - let mut prev: Option<&[u8]> = None; - for chunk in bytes.chunks_exact(UNSORTED_REC_LEN) { - let sh = &chunk[..SH_HEAD_KEY_LEN]; - if prev != Some(sh) { - n += 1; - prev = Some(sh); - } - } - n +struct ShardPool { + jobs: Mutex>, + err: Mutex>, } -fn read_unsorted_shard_bytes(path: &Path) -> Result, StoreError> { - match fs::read(path) { - Ok(b) => Ok(b), - Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(Vec::new()), - Err(e) => Err(StoreError::io(path, e)), - } +struct PackShardDone { + shard: usize, + keys: u64, + creates: u64, + fp_singles: u64, + elapsed: Duration, +} + +fn log_unsorted_pack_shard(n_shards: usize, published: u32, d: &PackShardDone) { + rbitcoin_log::info!( + "store: scripthash unsorted pack shard={:02x} keys={} creates={} fp_singles={} \ + shards={published}/{n_shards} elapsed={:?}", + d.shard, + d.keys, + d.creates, + d.fp_singles, + d.elapsed + ); } -fn pack_unsorted_shard( +fn pack_and_seal_unsorted_shard( table: &ScriptHashTable, - dir: &Path, + unsorted_dir: &Path, shard: usize, cancel: Option<&AtomicBool>, progress: &MaterializeProgress, -) -> Result { - let path = unsorted_shard_path(dir, shard); - let mut session = table.pack_shard_session(shard)?; - let t_loop = Instant::now(); - { - let mut bytes = read_unsorted_shard_bytes(&path)?; - sort_unique_unsorted_recs(&mut bytes)?; - session.reserve_pack_recs(unique_sh_key_count(&bytes)); - for (i, chunk) in bytes.chunks_exact(UNSORTED_REC_LEN).enumerate() { - if i & 0xfff == 0 && cancel.map(|c| c.load(Ordering::Relaxed)).unwrap_or(false) { - return Err(StoreError::Cancelled("scripthash unsorted shard pack")); - } - let (sh, fk) = decode_unsorted_rec(chunk)?; - session.push_sorted_fk(sh, fk)?; - progress.recs_packed.fetch_add(1, Ordering::Relaxed); - } - } - let loop_wall = t_loop.elapsed().as_nanos() as u64; - let pack_during_loop = session.pack_ns; - progress.merge_ns.fetch_add( - loop_wall.saturating_sub(pack_during_loop), - Ordering::Relaxed, - ); - let pack = session.finish_pack()?; + max_fk: &AtomicU64, +) -> Result { + check_cancel(cancel, "scripthash unsorted shard pack")?; + let t0 = Instant::now(); + let t_merge = Instant::now(); + let pack = pack_post_shard(table, unsorted_dir, shard, cancel)?; + progress + .merge_ns + .fetch_add(t_merge.elapsed().as_nanos() as u64, Ordering::Relaxed); progress.pack_ns.fetch_add(pack.pack_ns, Ordering::Relaxed); progress .body_flush_ns .fetch_add(pack.body_flush_ns, Ordering::Relaxed); progress.keys_packed.fetch_add(pack.keys, Ordering::Relaxed); - Ok(pack) + progress + .recs_packed + .fetch_add(pack.creates, Ordering::Relaxed); + let done = PackShardDone { + shard, + keys: pack.keys, + creates: pack.creates, + fp_singles: pack.fp_singles, + elapsed: t0.elapsed(), + }; + seal_shard(table, shard, pack, max_fk, progress)?; + unlink_post_shard(unsorted_dir, shard); + Ok(done) } -fn pack_and_seal_unsorted_shard( +#[cfg(test)] +pub(crate) fn pack_one_extract_shard( table: &ScriptHashTable, - dir: &Path, + unsorted_dir: &Path, shard: usize, - n_shards: usize, - cancel: Option<&AtomicBool>, - progress: &MaterializeProgress, - max_fk: &AtomicU64, ) -> Result<(), StoreError> { - let t0 = Instant::now(); - let pack = pack_unsorted_shard(table, dir, shard, cancel, progress)?; - let keys = pack.keys; - let creates = pack.creates; - seal_shard(table, shard, pack, max_fk, progress)?; - rbitcoin_log::info!( - "store: scripthash unsorted pack shard={shard:02x} keys={keys} creates={creates} \ - shards={}/{} elapsed={:?}", - progress.shards_published.load(Ordering::Relaxed), - n_shards, - t0.elapsed() - ); + let n = table.head_shard_count().max(1); + let progress = MaterializeProgress::new(); + let max_fk = AtomicU64::new(0); + let done = pack_and_seal_unsorted_shard(table, unsorted_dir, shard, None, &progress, &max_fk)?; + log_unsorted_pack_shard(n, progress.shards_published.load(Ordering::Relaxed), &done); Ok(()) } -/// Unique-sort each unsorted shard file in place and seal `head/NN` for unsealed shards. pub fn materialize_sh_from_unsorted( table: &ScriptHashTable, unsorted_dir: &Path, @@ -737,6 +1847,7 @@ pub fn materialize_sh_from_unsorted( cancel: Option<&AtomicBool>, ) -> Result { let n_shards = table.head_shard_count().max(1); + seal_mphf_from_keys(table, unsorted_dir, n_shards, cancel)?; let jobs: Vec = table .unsealed_main_shards() .into_iter() @@ -776,18 +1887,19 @@ pub fn materialize_sh_from_unsorted( let max_fk = &max_fk; if workers <= 1 { for shard in jobs { - if cancel.map(|c| c.load(Ordering::Relaxed)).unwrap_or(false) { - return Err(StoreError::Cancelled("scripthash unsorted shard pack")); - } - pack_and_seal_unsorted_shard( + let d = pack_and_seal_unsorted_shard( table, unsorted_dir, shard, - n_shards, cancel, progress, max_fk, )?; + log_unsorted_pack_shard( + n_shards, + progress.shards_published.load(Ordering::Relaxed), + &d, + ); } } else { let shared = Arc::new(ShardPool { @@ -816,12 +1928,17 @@ pub fn materialize_sh_from_unsorted( table, unsorted_dir, shard, - n_shards, cancel, progress, max_fk, ) { - Ok(()) => {} + Ok(d) => { + log_unsorted_pack_shard( + n_shards, + progress.shards_published.load(Ordering::Relaxed), + &d, + ); + } Err(e) => { *shared.err.lock().unwrap() = Some(e); break; @@ -862,7 +1979,6 @@ pub fn materialize_sh_from_unsorted( Ok(out) } -/// Collect (unless `DONE` is valid) then pack unsealed shards from Class A. pub fn materialize_sh_unsorted_from_class_a( store: &Store, collect_workers: usize, @@ -912,8 +2028,14 @@ pub fn materialize_sh_unsorted_from_class_a( any_sealed, cancel, )?; - - let mut mat = materialize_sh_from_unsorted(table, &dir, pack_workers, cancel)?; + let mut mat = materialize_sh_from_unsorted_from_txs( + table, + &store.txs, + &dir, + collect_workers, + pack_workers, + cancel, + )?; mat.max_fk = mat.max_fk.max(collected.last_fk); if table.unsealed_main_shards().is_empty() { clear_unsorted_shard_dir(&dir); @@ -922,224 +2044,368 @@ pub fn materialize_sh_unsorted_from_class_a( Ok(mat) } +/// Seal extract MPHF, scan Class A into postings, pack unsealed shards. +pub(crate) fn materialize_sh_from_unsorted_from_txs( + table: &ScriptHashTable, + txs: &TxTable, + unsorted_dir: &Path, + collect_workers: usize, + pack_workers: usize, + cancel: Option<&AtomicBool>, +) -> Result { + let n_shards = table.head_shard_count().max(1); + seal_mphf_from_keys(table, unsorted_dir, n_shards, cancel)?; + let any_sealed = table.unsealed_main_shards().len() < n_shards; + collect_posts_covering( + txs, + table, + unsorted_dir, + collect_workers, + any_sealed, + cancel, + )?; + materialize_sh_from_unsorted(table, unsorted_dir, pack_workers, cancel) +} + #[cfg(test)] mod tests { use super::*; + use crate::scripthash_layout::ShHeadKey; + use std::hash::{Hash, Hasher}; + + fn prefix_key(b0: u8) -> ShHeadKey { + let mut k = [0u8; KEY16_LEN]; + k[0] = b0; + k + } #[test] - fn materialize_stage_ns_are_populated() { - { - let dir = std::env::temp_dir().join(format!( - "rbitcoin-sh-stage-ns-{}-{}", - std::process::id(), - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_nanos() - )); - let _ = std::fs::remove_dir_all(&dir); - std::fs::create_dir_all(&dir).unwrap(); - let table = crate::scripthash::ScriptHashTable::create_tiny(&dir).unwrap(); - let n_shards = table.head_shard_count(); - let udir = dir.join("unsorted"); - fs::create_dir_all(&udir).unwrap(); - let mut bytes = Vec::new(); - for i in 0..32u64 { - let mut sh = [0u8; 32]; - sh[0] = i as u8; - bytes.extend_from_slice(&encode_unsorted_rec(&sh, Fk(i * 10 + 1))); - bytes.extend_from_slice(&encode_unsorted_rec(&sh, Fk(i * 10 + 2))); - } - fs::write(unsorted_shard_path(&udir, 0), &bytes).unwrap(); - for shard in 1..n_shards { - fs::write(unsorted_shard_path(&udir, shard), []).unwrap(); - } - let mat = materialize_sh_from_unsorted(&table, &udir, 1, None).unwrap(); - assert!(mat.creates >= 16); - assert!( - mat.pack_ns > 0 && mat.body_flush_ns > 0 && mat.mphf_ns > 0, - "stage ns must be real merge={} pack={} mphf={} body={}", - mat.merge_ns, - mat.pack_ns, - mat.mphf_ns, - mat.body_flush_ns - ); - assert_eq!( - mat.head_fill_ns, 0, - "pack-only materialize must not alias head_fill to mphf" - ); - let _ = std::fs::remove_dir_all(&dir); - } + fn key16_identity_hasher_is_xor_of_u64_halves() { + let mut k = [0u8; KEY16_LEN]; + k[0..8].copy_from_slice(&0x0102_0304_0506_0708u64.to_le_bytes()); + k[8..16].copy_from_slice(&0x1112_1314_1516_1718u64.to_le_bytes()); + let mut h = Key16IdentityHasher::default(); + k.hash(&mut h); + assert_eq!(h.finish(), 0x0102_0304_0506_0708u64 ^ 0x1112_1314_1516_1718); + let mut h2 = Key16IdentityHasher::default(); + k.hash(&mut h2); + assert_eq!(h.finish(), h2.finish()); + } + + #[test] + fn key16_identity_try_insert_first_second_third_is_pack8_word() { + let mut map = Key16PackMap::default(); + let k = prefix_key(1); + insert_key_pack(&mut map, k, Fk(10)); + assert_eq!(map.get(&k).copied(), Some(10)); + insert_key_pack(&mut map, k, Fk(11)); + assert_eq!(map.get(&k).copied(), Some(0)); + insert_key_pack(&mut map, k, Fk(12)); + assert_eq!(map.get(&k).copied(), Some(0), "third hit must not write"); + assert_eq!(KEYS_SPILL_MAGIC, b"SHKSP01\0"); + assert_eq!(KEYS_COLLECT_BYTES_PER_KEY, 64); + assert_eq!(SH_EXTRACT_WORKER_RAM_BYTES, 3 << 29); } #[test] - fn unsorted_collect_workers_are_n_cores() { + fn worker_fk_spans_are_contiguous_covering_and_disjoint() { + assert_eq!( + worker_fk_spans(1, 100, 4), + vec![(1, 25), (26, 50), (51, 75), (76, 100)] + ); + assert_eq!( + worker_fk_spans(1, 10, 3), + vec![(1, 3), (4, 6), (7, 10)], + "remainder on the last span" + ); + assert_eq!(worker_fk_spans(1, 5, 1), vec![(1, 5)]); assert_eq!( - unsorted_collect_workers(), - crate::sorted_run::logical_cpus(), - "collect workers follow nCPU (no RAM cap / env)" + worker_fk_spans(1, 3, 10), + vec![(1, 1), (2, 2), (3, 3)], + "n_workers > span clamps to one fk per worker" ); + assert!(worker_fk_spans(5, 4, 4).is_empty()); + assert_eq!(worker_fk_spans(1, 8, 0), vec![(1, 8)]); } #[test] - fn unsorted_pack_workers_use_2gib_budget() { + fn flush_collect_tally_matches_per_item_and_skips_zeros() { + let recs = AtomicU64::new(0); + let hits = AtomicU64::new(0); + let per = [AtomicU64::new(0), AtomicU64::new(0)]; + let mut one = CollectTally::new(2); + for _ in 0..3 { + one.recs = one.recs.saturating_add(1); + one.hits = one.hits.saturating_add(1); + one.per[0] = one.per[0].saturating_add(1); + flush_collect_tally(&mut one, &recs, Some(&hits), Some(&per)); + } + let batch_recs = AtomicU64::new(0); + let batch_hits = AtomicU64::new(0); + let batch_per = [AtomicU64::new(0), AtomicU64::new(0)]; + let mut batch = CollectTally { + recs: 3, + hits: 3, + per: vec![3, 0], + }; + flush_collect_tally(&mut batch, &batch_recs, Some(&batch_hits), Some(&batch_per)); + assert_eq!( + recs.load(Ordering::Relaxed), + batch_recs.load(Ordering::Relaxed) + ); + assert_eq!( + hits.load(Ordering::Relaxed), + batch_hits.load(Ordering::Relaxed) + ); assert_eq!( - crate::sorted_run::workers_for_free_ram(8, 3 << 30, SH_UNSORTED_PACK_RAM_BYTES), - 1, - "3 GiB free / 2 GiB per unsorted pack worker" + per[0].load(Ordering::Relaxed), + batch_per[0].load(Ordering::Relaxed) + ); + assert_eq!(per[1].load(Ordering::Relaxed), 0); + let snap = ( + batch_recs.load(Ordering::Relaxed), + batch_hits.load(Ordering::Relaxed), + batch_per[0].load(Ordering::Relaxed), ); + flush_collect_tally(&mut batch, &batch_recs, Some(&batch_hits), Some(&batch_per)); assert_eq!( - crate::sorted_run::workers_for_free_ram(16, 8 << 30, SH_UNSORTED_PACK_RAM_BYTES), - 4, - "8 GiB free → 4 unsorted pack workers" + snap, + ( + batch_recs.load(Ordering::Relaxed), + batch_hits.load(Ordering::Relaxed), + batch_per[0].load(Ordering::Relaxed), + ) ); } - fn rec_bytes(sh0: u8, fk: u64) -> [u8; UNSORTED_REC_LEN] { - encode_unsorted_rec(&[sh0; 32], Fk(fk)) + #[test] + fn scanned_fks_count_finished_spans_not_max_fk() { + let spans = worker_fk_spans(1, 100, 4); + assert_eq!(spans[3], (76, 100)); + let scanned = AtomicU64::new(0); + let one = note_scanned_fks(&scanned, spans[3].0, spans[3].0); + assert_eq!(one, 1, "finishing one fk of the late span is not fk 76"); + note_scanned_fks(&scanned, spans[0].0, spans[0].1); + assert_eq!(scanned.load(Ordering::Relaxed), 1 + 25); } - fn recs_of(bytes: &[u8]) -> Vec<([u8; 32], u64)> { - bytes - .chunks_exact(UNSORTED_REC_LEN) - .map(|c| { - let mut sh = [0u8; 32]; - sh[..SH_HEAD_KEY_LEN].copy_from_slice(&c[..SH_HEAD_KEY_LEN]); - ( - sh, - u64::from_le_bytes(c[SH_HEAD_KEY_LEN..].try_into().unwrap()), - ) - }) - .collect() + #[test] + fn keys_spill_channel_is_one_slot() { + let (tx, rx) = keys_spill_channel(); + let mut first = Key16PackMap::default(); + insert_key_pack(&mut first, prefix_key(1), Fk(1)); + tx.send((0, first)).unwrap(); + let mut second = Key16PackMap::default(); + insert_key_pack(&mut second, prefix_key(2), Fk(2)); + match tx.try_send((0, second)) { + Err(mpsc::TrySendError::Full(_)) => {} + other => panic!("second spill must block on the one-slot queue, got {other:?}"), + } + assert_eq!(rx.recv().unwrap().0, 0); } - fn prefix_key(b: u8) -> [u8; 32] { - let mut k = [0u8; 32]; - k[..SH_HEAD_KEY_LEN].fill(b); - k + #[test] + fn keys_spill_writer_folds_two_senders() { + let dir = crate::testutil::TempDir::labeled("sh-spill-writer").expect("temp"); + let seqs = [AtomicU32::new(0)]; + let err = Mutex::new(None); + let (tx, rx) = keys_spill_channel(); + std::thread::scope(|scope| { + scope.spawn(|| keys_spill_writer(rx, dir.path(), &seqs, &err)); + let tx_b = tx.clone(); + scope.spawn(move || { + let mut map = Key16PackMap::default(); + insert_key_pack(&mut map, prefix_key(1), Fk(1)); + send_keys_spill(&tx_b, 0, map).unwrap(); + }); + let mut map = Key16PackMap::default(); + insert_key_pack(&mut map, prefix_key(2), Fk(9)); + send_keys_spill(&tx, 0, map).unwrap(); + drop(tx); + }); + assert!(err.lock().unwrap().is_none()); + let got = load_keys_shard_map(dir.path(), 0).unwrap(); + assert_eq!(got.get(&prefix_key(1)).copied(), Some(1)); + assert_eq!(got.get(&prefix_key(2)).copied(), Some(9)); } - fn tmp_sink() -> (PathBuf, UnsortedShardSink) { - let dir = std::env::temp_dir().join(format!( - "rbitcoin-sh-unsink-{}-{}", - std::process::id(), - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_nanos() - )); - fs::create_dir_all(&dir).unwrap(); - let path = dir.join("00"); - let file = OpenOptions::new() - .create(true) - .write(true) - .truncate(true) - .open(&path) - .unwrap(); - let sink = UnsortedShardSink { - path: path.clone(), - file, - write: Mutex::new(UnsortedShardWrite { - cursor: 0, - allocated: 0, - }), - }; - (dir, sink) + #[test] + fn leftover_keys_file_refuses_to_open() { + let dir = crate::testutil::TempDir::labeled("sh-leftover-file").expect("temp"); + std::fs::create_dir_all(dir.join(KEYS_SUBDIR)).unwrap(); + let path = unsorted_keys_path(dir.path(), 0); + crate::file::write_synced_tmp_rename(&path, b"not-a-spill-dir").unwrap(); + match load_keys_shard_map(dir.path(), 0) { + Err(StoreError::Corrupt(m)) => { + assert!(m.contains("wipe store/scripthash.unsorted"), "{m}"); + } + other => panic!("must refuse leftover keys file, got {other:?}"), + } + assert!(leftover_legacy_unsorted(dir.path())); } #[test] - fn unsorted_flush_appends_in_offset_order() { - let (dir, sink) = tmp_sink(); - let mut a = rec_bytes(1, 1).to_vec(); - let mut b = rec_bytes(2, 2).to_vec(); - flush_unsorted_buf(&sink, &mut a).unwrap(); - flush_unsorted_buf(&sink, &mut b).unwrap(); - let bytes = fs::read(&sink.path).unwrap(); - assert_eq!( - recs_of(&bytes), - vec![(prefix_key(1), 1), (prefix_key(2), 2)] - ); - let _ = fs::remove_dir_all(&dir); + fn two_spills_fold_repeat_key_to_zero_without_sort() { + let dir = crate::testutil::TempDir::labeled("sh-two-spill").expect("temp"); + let a = prefix_key(1); + let b = prefix_key(2); + write_keys_spill_entries(dir.path(), 0, 0, &[(b, Some(Fk(5))), (a, Some(Fk(1)))]).unwrap(); + write_keys_spill_entries(dir.path(), 0, 1, &[(a, Some(Fk(9)))]).unwrap(); + assert!(unsorted_keys_spill_path(dir.path(), 0, 0).is_file()); + assert!(unsorted_keys_spill_path(dir.path(), 0, 1).is_file()); + assert!(unsorted_keys_path(dir.path(), 0).is_dir()); + let map = load_keys_shard_map(dir.path(), 0).unwrap(); + assert_eq!(map.get(&a).copied(), Some(0), "cross-spill repeat is multi"); + assert_eq!(map.get(&b).copied(), Some(5)); + assert_eq!(map.len(), 2); } #[test] - fn unsorted_flush_concurrent_workers_do_not_tear_recs() { - let (dir, sink) = tmp_sink(); - std::thread::scope(|scope| { - for t in 0u8..4 { - let sink = &sink; - scope.spawn(move || { - let mut buf = Vec::new(); - for i in 0..1000u64 { - buf.extend_from_slice(&rec_bytes(t, t as u64 * 1000 + i + 1)); - if buf.len() >= UNSORTED_REC_LEN * 17 { - flush_unsorted_buf(sink, &mut buf).unwrap(); - } - } - flush_unsorted_buf(sink, &mut buf).unwrap(); - }); + fn keys_spill_magic_wrong_refuses() { + let dir = crate::testutil::TempDir::labeled("sh-bad-magic").expect("temp"); + std::fs::create_dir_all(unsorted_keys_path(dir.path(), 0)).unwrap(); + let path = unsorted_keys_spill_path(dir.path(), 0, 0); + crate::file::write_synced_tmp_rename(&path, b"SHKEYS02........").unwrap(); + match load_keys_shard_map(dir.path(), 0) { + Err(StoreError::Corrupt(m)) => { + assert!(m.contains("wipe store/scripthash.unsorted"), "{m}"); } - }); - let bytes = fs::read(&sink.path).unwrap(); - assert_eq!(bytes.len(), 4 * 1000 * UNSORTED_REC_LEN); - for chunk in bytes.chunks_exact(UNSORTED_REC_LEN) { - let fk = u64::from_le_bytes(chunk[SH_HEAD_KEY_LEN..].try_into().unwrap()); - assert!(fk >= 1, "every rec must be a full 24-byte create"); + other => panic!("wrong magic must refuse, got {other:?}"), } - let _ = fs::remove_dir_all(&dir); } #[test] - fn sort_unique_unsorted_recs_orders_fk_numerically_and_compacts() { - let mut bytes = Vec::new(); - bytes.extend_from_slice(&rec_bytes(7, 256)); - bytes.extend_from_slice(&rec_bytes(7, 2)); - bytes.extend_from_slice(&rec_bytes(7, 2)); - bytes.extend_from_slice(&rec_bytes(7, 0)); - bytes.extend_from_slice(&rec_bytes(8, 1)); - bytes.extend_from_slice(&rec_bytes(6, 9)); - sort_unique_unsorted_recs(&mut bytes).unwrap(); + fn spill_largest_keys_writes_fattest_and_spills_while_over() { + let dir = crate::testutil::TempDir::labeled("sh-spill-largest").expect("temp"); + let mut maps: Vec = (0..3).map(|_| Key16PackMap::default()).collect(); + insert_key_pack(&mut maps[0], prefix_key(1), Fk(1)); + insert_key_pack(&mut maps[0], prefix_key(2), Fk(2)); + insert_key_pack(&mut maps[0], prefix_key(3), Fk(3)); + insert_key_pack(&mut maps[1], prefix_key(4), Fk(4)); + insert_key_pack(&mut maps[2], prefix_key(5), Fk(5)); + let seqs: Vec = (0..3).map(|_| AtomicU32::new(0)).collect(); + let budget = 2 * KEYS_COLLECT_BYTES_PER_KEY; + spill_largest_keys_while_over(&mut maps, budget, |si, map| { + spill_shard_map(dir.path(), si, &seqs, map) + }) + .unwrap(); + assert!(maps[0].is_empty(), "fattest shard spilled first"); + assert!( + maps[1].is_empty() ^ maps[2].is_empty(), + "second while-over spill empties exactly one remaining shard" + ); + let leftover = if maps[1].is_empty() { 2 } else { 1 }; + assert_eq!(maps[leftover].len(), 1, "one shard stays under budget"); + assert!(unsorted_keys_spill_path(dir.path(), 0, 0).is_file()); + let n_files = [0, 1, 2] + .iter() + .filter(|si| unsorted_keys_spill_path(dir.path(), **si, 0).is_file()) + .count(); + assert_eq!(n_files, 2); + let got0 = load_keys_shard_map(dir.path(), 0).unwrap(); + assert_eq!(got0.len(), 3); + } + + #[test] + fn multi_fuse8_contains_inserted_mix64() { + let k = prefix_key(9); + let mixed = mix_key16(&k); + let fuse = SealedFuse8::build(&[mixed]).unwrap(); + assert!(fuse.contains(mixed)); + } + + #[test] + fn sh_extract_workers_cap_at_1_5gib() { + assert_eq!(SH_EXTRACT_WORKER_RAM_BYTES, 3 << 29); assert_eq!( - recs_of(&bytes), - vec![ - (prefix_key(6), 9), - (prefix_key(7), 2), - (prefix_key(7), 256), - (prefix_key(8), 1), - ], - "numeric fk order (256 after 2), drop null and duplicate (sh,fk)" + crate::sorted_run::workers_for_free_ram(8, 3 << 30, SH_EXTRACT_WORKER_RAM_BYTES), + 2 + ); + assert_eq!( + crate::sorted_run::workers_for_free_ram(16, 24 << 30, SH_EXTRACT_WORKER_RAM_BYTES), + 16 ); - assert_eq!(unique_sh_key_count(&bytes), 3); } #[test] - fn sort_unique_unsorted_recs_rejects_torn_length() { - let err = sort_unique_unsorted_recs(&mut vec![0u8; 39]).unwrap_err(); - assert!( - matches!(err, StoreError::Corrupt(m) if m.contains("multiple of 24")), - "got {err}" + fn class_a_scan_chunk_is_loc_fold_grain() { + assert_eq!( + CLASS_A_CHUNK_FKS, + 1 << 16, + "inner loc/body batch inside a static worker span, not a steal grain" + ); + assert_eq!( + chunk_ranges(1, CLASS_A_CHUNK_FKS + 1, CLASS_A_CHUNK_FKS), + vec![ + (1, CLASS_A_CHUNK_FKS), + (CLASS_A_CHUNK_FKS + 1, CLASS_A_CHUNK_FKS + 1) + ] + ); + assert_eq!( + chunk_ranges(1, 70_000, CLASS_A_CHUNK_FKS), + vec![(1, CLASS_A_CHUNK_FKS), (CLASS_A_CHUNK_FKS + 1, 70_000)] ); } #[test] - fn unsorted_rec_is_16_byte_head_prefix_plus_fk() { - let mut sh = [0u8; 32]; - sh[..16].fill(0x11); - sh[16..].fill(0x22); - let rec = encode_unsorted_rec(&sh, Fk(0x0102_0304_0506_0708)); - assert_eq!(UNSORTED_REC_LEN, 24); - assert_eq!(rec.len(), 24); - assert_eq!(&rec[..16], &[0x11; 16]); - assert_eq!(&rec[16..], &0x0102_0304_0506_0708u64.to_le_bytes()); - assert!( - rec.iter().all(|&b| b != 0x22), - "must not store the trailing 16 hash bytes" + fn keys_done_magic_is_shkeys02() { + assert_eq!(KEYS_DONE_MAGIC, b"SHKEYS02"); + assert_eq!(POST_DONE_MAGIC, b"SHPOST02"); + assert_eq!(POSTS_SPILL_MAGIC, b"SHPST01\0"); + assert_eq!(POST_MAP_KEY_BYTES, 80); + assert_eq!(POST_MAP_FK_BYTES, 8); + } + + #[test] + fn spill_largest_posts_writes_fattest_shard() { + let dir = crate::testutil::TempDir::labeled("sh-post-largest").expect("temp"); + let mut maps: Vec = (0..2).map(|_| PostPackMap::default()).collect(); + insert_post_fk(&mut maps[0], prefix_key(1), 1); + insert_post_fk(&mut maps[0], prefix_key(1), 2); + insert_post_fk(&mut maps[1], prefix_key(2), 3); + let seqs: Vec = (0..2).map(|_| AtomicU32::new(0)).collect(); + let fat = maps[0].estimated_bytes() as u64; + spill_largest_posts_while_over(&mut maps, fat.saturating_sub(1), |si, map| { + spill_post_shard_map(dir.path(), si, &seqs, map) + }) + .unwrap(); + assert!(maps[0].is_empty()); + assert_eq!(maps[1].map.len(), 1); + let folded = load_post_shard_map(dir.path(), 0).unwrap(); + assert_eq!( + folded.map.get(&prefix_key(1)).map(|v| v.as_slice()), + Some(&[1u64, 2][..]) ); } #[test] - fn unsorted_done_magic_is_v3() { - assert_eq!(UNSORTED_DONE_MAGIC, b"SHUNSRT3"); + fn leftover_shunsrt3_done_is_not_ok() { + let dir = std::env::temp_dir().join(format!( + "rbitcoin-sh-done-v3-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let n_shards = 2usize; + let mut buf = Vec::new(); + buf.extend_from_slice(b"SHUNSRT3"); + buf.extend_from_slice(&(n_shards as u32).to_le_bytes()); + buf.extend_from_slice(&1u64.to_le_bytes()); + buf.extend_from_slice(&0u64.to_le_bytes()); + buf.extend_from_slice(&0u64.to_le_bytes()); + fs::write(dir.join("DONE"), &buf).unwrap(); + fs::write(unsorted_shard_path(&dir, 0), [0u8; 24]).unwrap(); + fs::write(unsorted_shard_path(&dir, 1), []).unwrap(); + assert!( + unsorted_done_last_fk(&dir, n_shards).is_none(), + "SHUNSRT3 must not count as DONE.keys" + ); + assert!(leftover_legacy_unsorted(&dir)); + let _ = fs::remove_dir_all(&dir); } #[test] @@ -1173,36 +2439,233 @@ mod tests { #[test] fn plan_unsorted_collect_appends_only_when_no_shards() { assert!(matches!( - plan_unsorted_collect(None, 10, false), + plan_unsorted_collect(None, 10, false, true), UnsortedCollectAction::Full )); assert!(matches!( - plan_unsorted_collect(Some(10), 10, false), + plan_unsorted_collect(Some(10), 10, false, true), UnsortedCollectAction::Skip )); assert!(matches!( - plan_unsorted_collect(Some(4), 10, false), + plan_unsorted_collect(Some(4), 10, false, true), UnsortedCollectAction::Append { first: 5, last: 10 } )); assert!(matches!( - plan_unsorted_collect(Some(4), 10, true), + plan_unsorted_collect(Some(4), 10, true, true), UnsortedCollectAction::Skip )); + assert!(matches!( + plan_unsorted_collect(Some(4), 10, false, false), + UnsortedCollectAction::Full + )); + } + + #[test] + fn keys_spill_delta_roundtrip_first_fk_order() { + let mut map = Key16PackMap::default(); + insert_key_pack(&mut map, prefix_key(3), Fk(100)); + insert_key_word(&mut map, prefix_key(1), 0); + insert_key_pack(&mut map, prefix_key(2), Fk(20)); + let bytes = encode_keys_spill(&map); + assert_eq!(&bytes[0..8], KEYS_SPILL_MAGIC); + let mut got = Key16PackMap::default(); + fold_keys_spill_bytes(&mut got, &bytes).unwrap(); + assert_eq!(got.get(&prefix_key(1)).copied(), Some(0)); + assert_eq!(got.get(&prefix_key(2)).copied(), Some(20)); + assert_eq!(got.get(&prefix_key(3)).copied(), Some(100)); + } + + #[test] + fn reused_script_two_spans_one_spill_rec_under_cap() { + let mut map = Key16PackMap::default(); + let k = prefix_key(7); + insert_key_pack(&mut map, k, Fk(1)); + insert_key_pack(&mut map, k, Fk(65_536)); + assert_eq!(map.len(), 1); + assert_eq!(map.get(&k).copied(), Some(0)); + let bytes = encode_keys_spill(&map); + let n_multi = u32::from_le_bytes(bytes[8..12].try_into().unwrap()); + let n_single = u32::from_le_bytes(bytes[12..16].try_into().unwrap()); + assert_eq!(n_multi, 1, "one rec while the map has not filled"); + assert_eq!(n_single, 0); + } + + #[test] + fn hashmap_iteration_order_seals_valid_mphf() { + let mut map = Key16PackMap::default(); + insert_key_pack(&mut map, prefix_key(1), Fk(10)); + insert_key_pack(&mut map, prefix_key(2), Fk(20)); + insert_key_word(&mut map, prefix_key(3), 0); + let mut recs = Vec::with_capacity(map.len()); + let mut mixed = Vec::with_capacity(map.len()); + for (k, w) in &map { + mixed.push(mix_key16(k)); + recs.push((*k, *w)); + } + let dir = crate::testutil::TempDir::labeled("sh-map-mphf").expect("temp"); + let base = dir.join("00"); + MphfHead::write_pack8_mixed(&base, &recs, &mixed).unwrap(); + let h = MphfHead::open(&base).unwrap(); + assert_eq!( + h.get(&prefix_key(1)).unwrap().unwrap(), + crate::scripthash_layout::ShHeadValue::inline_one(Fk(10)) + ); + assert_eq!( + h.get(&prefix_key(2)).unwrap().unwrap(), + crate::scripthash_layout::ShHeadValue::inline_one(Fk(20)) + ); + assert_eq!( + h.get(&prefix_key(3)).unwrap().unwrap(), + crate::scripthash_layout::ShHeadValue::Empty + ); + } + + #[test] + fn insert_post_fk_out_of_order_stays_strict_and_roundtrips() { + let mut map = PostPackMap::default(); + let k = prefix_key(4); + insert_post_fk(&mut map, k, 3); + insert_post_fk(&mut map, k, 1); + insert_post_fk(&mut map, k, 3); + assert_eq!(map.map.get(&k).map(|v| v.as_slice()), Some(&[1u64, 3][..])); + assert_eq!(map.n_fks, 2); + let bytes = encode_posts_spill(&map); + let mut round = PostPackMap::default(); + fold_posts_spill_bytes(&mut round, &bytes).unwrap(); + assert_eq!( + round.map.get(&k).map(|v| v.as_slice()), + Some(&[1u64, 3][..]) + ); + } + + #[test] + fn fold_post_spills_merges_sorted_runs() { + let dir = crate::testutil::TempDir::labeled("sh-post-merge-order").expect("temp"); + let k = prefix_key(8); + write_post_spill_entries(dir.path(), 0, 0, &[(k, &[1u64, 5][..])]).unwrap(); + write_post_spill_entries(dir.path(), 0, 1, &[(k, &[3u64, 5, 9][..])]).unwrap(); + let map = load_post_shard_map(dir.path(), 0).unwrap(); + assert_eq!( + map.map.get(&k).map(|v| v.as_slice()), + Some(&[1u64, 3, 5, 9][..]) + ); + assert_eq!(map.n_fks, 4); } #[test] - fn sort_unique_collapses_full_hashes_that_share_a_16_byte_prefix() { - let mut a = [0u8; 32]; - a[0] = 1; - a[31] = 9; - let mut b = [0u8; 32]; - b[0] = 1; - b[31] = 8; - let mut bytes = Vec::new(); - bytes.extend_from_slice(&encode_unsorted_rec(&a, Fk(3))); - bytes.extend_from_slice(&encode_unsorted_rec(&b, Fk(3))); - sort_unique_unsorted_recs(&mut bytes).unwrap(); - assert_eq!(bytes.len(), UNSORTED_REC_LEN); - assert_eq!(unique_sh_key_count(&bytes), 1); + fn posts_two_fks_one_spill_rec_under_cap() { + let mut map = PostPackMap::default(); + let k = prefix_key(7); + insert_post_fk(&mut map, k, 2); + insert_post_fk(&mut map, k, 2); + insert_post_fk(&mut map, k, 11); + assert_eq!(map.map.get(&k).map(|v| v.as_slice()), Some(&[2u64, 11][..])); + assert_eq!(map.n_fks, 2); + assert_eq!( + map.estimated_bytes(), + POST_MAP_KEY_BYTES + 2 * POST_MAP_FK_BYTES + ); + let bytes = encode_posts_spill(&map); + assert_eq!(&bytes[0..8], POSTS_SPILL_MAGIC); + let n_keys = u32::from_le_bytes(bytes[8..12].try_into().unwrap()); + assert_eq!(n_keys, 1); + let mut round = PostPackMap::default(); + fold_posts_spill_bytes(&mut round, &bytes).unwrap(); + assert_eq!( + round.map.get(&k).map(|v| v.as_slice()), + Some(&[2u64, 11][..]) + ); + } + + #[test] + fn tiny_post_budget_spills_then_flush() { + let dir = crate::testutil::TempDir::labeled("sh-tiny-post").expect("temp"); + let mut maps = vec![PostPackMap::default()]; + let seqs = [AtomicU32::new(0)]; + insert_post_fk(&mut maps[0], prefix_key(1), 1); + insert_post_fk(&mut maps[0], prefix_key(1), 2); + insert_post_fk(&mut maps[0], prefix_key(2), 3); + spill_largest_posts_while_over(&mut maps, 100, |si, map| { + spill_post_shard_map(dir.path(), si, &seqs, map) + }) + .unwrap(); + assert!(maps[0].is_empty()); + insert_post_fk(&mut maps[0], prefix_key(2), 4); + spill_all_post_maps(&mut maps, |si, map| { + spill_post_shard_map(dir.path(), si, &seqs, map) + }) + .unwrap(); + assert!(unsorted_post_spill_path(dir.path(), 0, 0).is_file()); + assert!(unsorted_post_spill_path(dir.path(), 0, 1).is_file()); + assert!(unsorted_post_path(dir.path(), 0).is_dir()); + let folded = load_post_shard_map(dir.path(), 0).unwrap(); + assert_eq!( + folded.map.get(&prefix_key(1)).map(|v| v.as_slice()), + Some(&[1u64, 2][..]) + ); + assert_eq!( + folded.map.get(&prefix_key(2)).map(|v| v.as_slice()), + Some(&[3u64, 4][..]) + ); + } + + #[test] + fn two_post_spills_fold_same_key() { + let dir = crate::testutil::TempDir::labeled("sh-post-fold").expect("temp"); + let k = prefix_key(9); + write_post_spill_entries(dir.path(), 0, 0, &[(k, &[1u64, 3][..])]).unwrap(); + write_post_spill_entries(dir.path(), 0, 1, &[(k, &[3u64, 5][..])]).unwrap(); + let map = load_post_shard_map(dir.path(), 0).unwrap(); + assert_eq!( + map.map.get(&k).map(|v| v.as_slice()), + Some(&[1u64, 3, 5][..]) + ); + } + + #[test] + fn leftover_post_file_refuses_to_open() { + let dir = crate::testutil::TempDir::labeled("sh-leftover-post").expect("temp"); + std::fs::create_dir_all(dir.join(POST_SUBDIR)).unwrap(); + let path = unsorted_post_path(dir.path(), 0); + crate::file::write_synced_tmp_rename(&path, b"not-a-spill-dir").unwrap(); + match load_post_shard_map(dir.path(), 0) { + Err(StoreError::Corrupt(m)) => { + assert!(m.contains("wipe store/scripthash.unsorted"), "{m}"); + } + other => panic!("must refuse leftover post file, got {other:?}"), + } + assert!(leftover_legacy_unsorted(dir.path())); + assert!(leftover_post_file(dir.path())); + } + + #[test] + fn leftover_post_spill_wrong_magic_refuses() { + let dir = crate::testutil::TempDir::labeled("sh-post-bad-magic").expect("temp"); + std::fs::create_dir_all(unsorted_post_path(dir.path(), 0)).unwrap(); + let path = unsorted_post_spill_path(dir.path(), 0, 0); + crate::file::write_synced_tmp_rename(&path, b"SHPOST02........").unwrap(); + match load_post_shard_map(dir.path(), 0) { + Err(StoreError::Corrupt(m)) => { + assert!(m.contains("wipe store/scripthash.unsorted"), "{m}"); + } + other => panic!("wrong magic must refuse, got {other:?}"), + } + } + + #[test] + fn posts_spill_fallocate_padding_refuses() { + let mut map = PostPackMap::default(); + insert_post_fk(&mut map, prefix_key(1), 7); + let mut bytes = encode_posts_spill(&map); + bytes.extend_from_slice(&[0u8; 64]); + match fold_posts_spill_bytes(&mut PostPackMap::default(), &bytes) { + Err(StoreError::Corrupt(m)) => { + assert!( + m.contains("trailing bytes"), + "padding after SHPST01 must not scan as recs, got {m}" + ); + } + other => panic!("fallocate padding must refuse, got {other:?}"), + } } } diff --git a/crates/rbitcoin-store/src/scripthash_mphf.rs b/crates/rbitcoin-store/src/scripthash_mphf.rs index 205a8cc95..e6e634c35 100644 --- a/crates/rbitcoin-store/src/scripthash_mphf.rs +++ b/crates/rbitcoin-store/src/scripthash_mphf.rs @@ -6,7 +6,6 @@ use crate::bdz::BdzMphf; use crate::error::StoreError; -use crate::fuse8_filter::fuse_key_from_mixed; use crate::io_handle::IoHandle; use crate::scripthash_layout::{pack8, unpack8, ShHeadKey, ShHeadValue, SH_HEAD_KEY_LEN}; use std::fs::{File, OpenOptions}; @@ -38,21 +37,12 @@ fn sidecar(base: &Path, ext: &str) -> PathBuf { } pub fn mix_key16(key: &ShHeadKey) -> u64 { - let mut pad = [0u8; 32]; - pad[..SH_HEAD_KEY_LEN].copy_from_slice(key); - fuse_key_from_mixed(&pad) + u64::from_le_bytes(key[0..8].try_into().expect("key16 half")) + ^ u64::from_le_bytes(key[8..16].try_into().expect("key16 half")) } -fn mix64_keys_unique(recs: &[(ShHeadKey, u64)]) -> Result, StoreError> { - let keys: Vec = recs.iter().map(|(k, _)| mix_key16(k)).collect(); - if keys.len() >= 2 { - let mut sorted = keys.clone(); - sorted.sort_unstable(); - if sorted.windows(2).any(|w| w[0] == w[1]) { - return Err(StoreError::Corrupt("sh mphf: mix64 collision")); - } - } - Ok(keys) +pub(crate) fn mix64_keys(recs: &[(ShHeadKey, u64)]) -> Vec { + recs.iter().map(|(k, _)| mix_key16(k)).collect() } impl MphfHead { @@ -91,13 +81,21 @@ impl MphfHead { pub fn write_pack8( base: impl AsRef, recs: &[(ShHeadKey, u64)], + ) -> Result { + let keys = mix64_keys(recs); + Self::write_pack8_mixed(base, recs, &keys) + } + + pub(crate) fn write_pack8_mixed( + base: impl AsRef, + recs: &[(ShHeadKey, u64)], + keys: &[u64], ) -> Result { let base = base.as_ref(); if let Some(parent) = base.parent() { let _ = std::fs::create_dir_all(parent); } - let keys = mix64_keys_unique(recs)?; - let mphf = BdzMphf::build_compact(&keys)?; + let mphf = BdzMphf::build_compact(keys)?; let n = recs.len(); let mut val = vec![0u8; n.saturating_mul(8)]; let mut tags = vec![0u8; n.saturating_mul(8)]; @@ -125,6 +123,48 @@ impl MphfHead { Self::open(base) } + /// MPHF slot for a key known to be in the set (no tag pread). + pub fn slot_for_key16(&self, key: &ShHeadKey) -> Result { + if self.mphf.n() == 0 { + return Err(StoreError::Corrupt("sh mphf: empty shard has key")); + } + self.mphf.index(mix_key16(key)) + } + + /// Overwrite pack8 words by MPHF slot (pass-2 pack; does not rebuild BDZ). + /// + /// Every slot is checked before any write. `.val` up to 512 MiB is patched + /// as one image; larger files use 1 MiB windows. + pub fn rewrite_val_slots(&self, words: &[(u32, u64)]) -> Result<(), StoreError> { + let n = self.mphf.n(); + for &(slot, _) in words { + if slot >= n { + return Err(StoreError::Corrupt("sh mphf: slot OOB")); + } + } + let len = u64::from(n).saturating_mul(8); + let path = val_path(&self.base); + if words.is_empty() || len == 0 { + self.val_file + .sync_data() + .map_err(|e| StoreError::io(&path, e))?; + return Ok(()); + } + if len <= VAL_REWRITE_FULL_MAX { + let mut buf = vec![0u8; len as usize]; + pread_file_exact(&self.val_file, 0, &mut buf).map_err(|e| StoreError::io(&path, e))?; + patch_val_image(&mut buf, 0, words); + pwrite_file(&self.val_file, 0, &buf).map_err(|e| StoreError::io(&path, e))?; + } else { + rewrite_val_windows(&self.val_file, words, VAL_REWRITE_WINDOW) + .map_err(|e| StoreError::io(&path, e))?; + } + self.val_file + .sync_data() + .map_err(|e| StoreError::io(&path, e))?; + Ok(()) + } + pub fn open(base: impl AsRef) -> Result { let base = base.as_ref().to_path_buf(); let mp = mphf_path(&base); @@ -217,6 +257,68 @@ impl MphfHead { } } +/// Patch a whole `.val` in RAM up to this size. Above it, windowed RMW. +const VAL_REWRITE_FULL_MAX: u64 = 512 << 20; +/// Byte cap of one read-modify-write span on a large `.val`. +const VAL_REWRITE_WINDOW: u64 = 1 << 20; + +fn patch_val_image(buf: &mut [u8], base_slot: u32, words: &[(u32, u64)]) { + for &(slot, w) in words { + let rel = (slot - base_slot) as usize * 8; + buf[rel..rel + 8].copy_from_slice(&w.to_le_bytes()); + } +} + +/// Inclusive slot spans whose byte width is at most `window_bytes`. +/// `words` must be sorted by slot. Later duplicates of a slot stay in the +/// same span so the caller applies them in order. +fn val_window_spans(words: &[(u32, u64)], window_bytes: u64) -> Vec<(u32, u32)> { + let mut out = Vec::new(); + let mut i = 0usize; + while i < words.len() { + let start = words[i].0; + let mut end = start; + let mut j = i + 1; + while j < words.len() { + let slot = words[j].0; + let span = (u64::from(slot) - u64::from(start) + 1).saturating_mul(8); + if span > window_bytes.max(8) { + break; + } + end = slot; + j += 1; + } + out.push((start, end)); + i = j; + } + out +} + +fn rewrite_val_windows( + file: &File, + words: &[(u32, u64)], + window_bytes: u64, +) -> std::io::Result<()> { + let mut order: Vec = (0..words.len()).collect(); + order.sort_by_key(|&i| (words[i].0, i)); + let sorted: Vec<(u32, u64)> = order.iter().map(|&i| words[i]).collect(); + let mut at = 0usize; + for (start, end) in val_window_spans(&sorted, window_bytes) { + let mut next = at; + while next < sorted.len() && sorted[next].0 <= end { + next += 1; + } + let span = ((end - start) as usize + 1) * 8; + let off = u64::from(start) * 8; + let mut buf = vec![0u8; span]; + pread_file_exact(file, off, &mut buf)?; + patch_val_image(&mut buf, start, &sorted[at..next]); + pwrite_file(file, off, &buf)?; + at = next; + } + Ok(()) +} + fn pread_file_exact(file: &File, offset: u64, buf: &mut [u8]) -> std::io::Result<()> { let h = IoHandle::from_file(file); let mut done = 0usize; @@ -359,15 +461,145 @@ mod tests { } #[test] - fn sh_mphf_duplicate_key16_is_mix64_collision() { + fn sh_mphf_write_keys_zero_val_then_rewrite_slots() { + let dir = tmp(); + let base = dir.join("00"); + let k1 = key(1); + let k2 = key(2); + let h = MphfHead::write_pack8(&base, &[(k1, 0), (k2, 0)]).unwrap(); + assert!(h.get(&k1).unwrap().unwrap().is_empty()); + let s1 = h.slot_for_key16(&k1).unwrap(); + let s2 = h.slot_for_key16(&k2).unwrap(); + let a = pack8(&ShHeadValue::inline_one(Fk(11))).unwrap(); + let b = pack8(&ShHeadValue::inline_one(Fk(22))).unwrap(); + h.rewrite_val_slots(&[(s1, a), (s2, b)]).unwrap(); + assert_eq!( + h.get(&k1).unwrap().unwrap(), + ShHeadValue::inline_one(Fk(11)) + ); + assert_eq!( + h.get(&k2).unwrap().unwrap(), + ShHeadValue::inline_one(Fk(22)) + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn sh_mphf_rewrite_val_edges_keep_middle_and_length() { + let dir = tmp(); + let base = dir.join("00"); + let keys: Vec = (1..=4).map(key).collect(); + let recs: Vec<(ShHeadKey, u64)> = keys + .iter() + .enumerate() + .map(|(i, k)| { + ( + *k, + pack8(&ShHeadValue::inline_one(Fk(100 + i as u64))).unwrap(), + ) + }) + .collect(); + let h = MphfHead::write_pack8(&base, &recs).unwrap(); + let n = h.mphf.n(); + assert_eq!(n, 4); + assert_eq!(std::fs::metadata(val_path(&base)).unwrap().len(), 32); + let mut by_slot = vec![None; n as usize]; + for (i, k) in keys.iter().enumerate() { + let slot = h.slot_for_key16(k).unwrap() as usize; + by_slot[slot] = Some((i, *k)); + } + let (_i0, k0) = by_slot[0].unwrap(); + let (_il, kl) = by_slot[n as usize - 1].unwrap(); + let mid_slot = (1..n as usize - 1).find(|s| by_slot[*s].is_some()).unwrap(); + let (im, km) = by_slot[mid_slot].unwrap(); + let w0 = pack8(&ShHeadValue::inline_one(Fk(7))).unwrap(); + let wl = pack8(&ShHeadValue::inline_one(Fk(9))).unwrap(); + h.rewrite_val_slots(&[(0, w0), (n - 1, wl)]).unwrap(); + assert_eq!(h.get(&k0).unwrap().unwrap(), ShHeadValue::inline_one(Fk(7))); + assert_eq!(h.get(&kl).unwrap().unwrap(), ShHeadValue::inline_one(Fk(9))); + assert_eq!( + h.get(&km).unwrap().unwrap(), + ShHeadValue::inline_one(Fk(100 + im as u64)) + ); + assert_eq!(std::fs::metadata(val_path(&base)).unwrap().len(), 32); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn sh_mphf_rewrite_val_oob_does_not_tear() { + let dir = tmp(); + let base = dir.join("00"); + let k1 = key(1); + let k2 = key(2); + let a = ShHeadValue::inline_one(Fk(11)); + let b = ShHeadValue::inline_one(Fk(22)); + let h = MphfHead::write_pack8(&base, &[(k1, pack8(&a).unwrap()), (k2, pack8(&b).unwrap())]) + .unwrap(); + let s1 = h.slot_for_key16(&k1).unwrap(); + let before = std::fs::read(val_path(&base)).unwrap(); + let err = h + .rewrite_val_slots(&[ + (s1, pack8(&ShHeadValue::inline_one(Fk(99))).unwrap()), + (99, 1), + ]) + .unwrap_err(); + assert!(matches!(err, StoreError::Corrupt(m) if m.contains("slot OOB"))); + assert_eq!(std::fs::read(val_path(&base)).unwrap(), before); + assert_eq!(h.get(&k1).unwrap().unwrap(), a); + assert_eq!(h.get(&k2).unwrap().unwrap(), b); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn val_window_spans_split_on_gap() { + let words = [(0u32, 1u64), (1, 2), (10, 3), (10, 4), (11, 5)]; + assert_eq!(val_window_spans(&words, 24), vec![(0, 1), (10, 11)]); + assert_eq!( + val_window_spans(&words, 8), + vec![(0, 0), (1, 1), (10, 10), (11, 11)] + ); + } + + #[test] + fn rewrite_val_windows_last_duplicate_wins() { + let dir = tmp(); + let path = dir.join("val"); + let mut raw = vec![0u8; 16 * 8]; + for slot in 0..16u32 { + let off = slot as usize * 8; + raw[off..off + 8].copy_from_slice(&(u64::from(slot) + 1).to_le_bytes()); + } + std::fs::write(&path, &raw).unwrap(); + let f = OpenOptions::new() + .read(true) + .write(true) + .open(&path) + .unwrap(); + rewrite_val_windows(&f, &[(0, 9), (15, 7), (0, 4), (2, 8)], 16).unwrap(); + let got = std::fs::read(&path).unwrap(); + assert_eq!(u64::from_le_bytes(got[0..8].try_into().unwrap()), 4); + assert_eq!(u64::from_le_bytes(got[8..16].try_into().unwrap()), 2); + assert_eq!(u64::from_le_bytes(got[16..24].try_into().unwrap()), 8); + assert_eq!(u64::from_le_bytes(got[24..32].try_into().unwrap()), 4); + assert_eq!( + u64::from_le_bytes(got[15 * 8..16 * 8].try_into().unwrap()), + 7 + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn sh_mphf_duplicate_key16_does_not_clone_sort_mix64() { let dir = tmp(); let base = dir.join("00"); let a = ShHeadValue::inline_one(Fk(11)); let recs = [(key(1), pack8(&a).unwrap()), (key(1), pack8(&a).unwrap())]; match MphfHead::write_pack8(&base, &recs) { - Err(StoreError::Corrupt(m)) if m.contains("mix64 collision") => {} - Err(e) => panic!("expected mix64 collision, got {e}"), - Ok(_) => panic!("expected mix64 collision"), + Err(StoreError::Corrupt(m)) if m.contains("mix64 collision") => { + panic!("mix64 clone-sort is not the collision path: {m}") + } + Err(StoreError::Corrupt(_)) | Ok(_) => {} + Err(e) => panic!("unexpected {e}"), } let _ = std::fs::remove_dir_all(&dir); } diff --git a/crates/rbitcoin-store/src/scripthash_tests.rs b/crates/rbitcoin-store/src/scripthash_tests.rs index ce2adf147..36b3a0389 100644 --- a/crates/rbitcoin-store/src/scripthash_tests.rs +++ b/crates/rbitcoin-store/src/scripthash_tests.rs @@ -1,9 +1,15 @@ use super::*; +use crate::fuse8_filter::SealedFuse8; use crate::scripthash_head::prefix_shard_of; -use crate::scripthash_layout::SH_HEAD_VALUE_LEN; +use crate::scripthash_layout::{head_key_from_full, ShHeadValue, SH_HEAD_VALUE_LEN}; use crate::scripthash_materialize::{ - materialize_sh_from_unsorted, unsorted_done_last_fk, unsorted_shard_path, UNSORTED_SHARD_DIR, + collect_unsorted_covering_txs, load_post_shard_entries, materialize_sh_from_unsorted, + materialize_sh_from_unsorted_from_txs, pack_one_extract_shard, seal_mphf_from_keys, + unsorted_done_last_fk, unsorted_keys_path, unsorted_mphf_base, unsorted_multi_fuse_path, + unsorted_post_path, unsorted_shard_path, write_keys_spill_entries, write_post_spill_entries, + UNSORTED_SHARD_DIR, }; +use crate::scripthash_mphf::{mix_key16, MphfHead}; use crate::scripthash_pages::{ sh_page_as_array, sh_page_as_array_mut, sh_page_extent, sh_page_set_extent, SH_PAGE_EXTENT_STREAM_MAX, SH_PAGE_SIZE, SH_PAGE_STREAM_MAX, @@ -2180,67 +2186,375 @@ fn class_a_coinbase( ) } -fn decode_unsorted_file(path: &std::path::Path) -> Vec { - let bytes = std::fs::read(path).unwrap(); - assert_eq!(bytes.len() % 24, 0, "unsorted file must be 24-byte recs"); - bytes - .chunks_exact(24) - .map(|c| { - let mut sh = [0u8; 32]; - sh[..16].copy_from_slice(&c[..16]); - let fk = Fk(u64::from_le_bytes(c[16..24].try_into().unwrap())); - ScriptHashRecord::from_fk(sh, fk) - }) +fn decode_post_keys(dir: &std::path::Path, shard: usize) -> Vec<[u8; 16]> { + load_post_shard_entries(dir, shard) + .unwrap() + .into_iter() + .map(|(k, _)| k) .collect() } #[test] -fn unsorted_collect_partitions_by_prefix_and_is_not_scripthash_sorted() { +fn unsorted_collect_unique_keys_per_shard() { { let dir = tmp(); let s = crate::Store::create_tiny(&dir).unwrap(); - let n_shards = 4usize; - let (low_script, high_script) = two_scripts_same_shard_reverse_hash(1, n_shards); - let mut txid_lo = [0u8; 32]; - txid_lo[0] = 1; - let mut txid_hi = [0u8; 32]; - txid_hi[0] = 2; - s.put_tx_full_batch_indexed(&[class_a_coinbase(txid_lo, high_script.clone())], true) + let n_shards = s.scripthash.head_shard_count(); + let script = vec![0x51]; + let mut txid_a = [0u8; 32]; + txid_a[0] = 1; + let mut txid_b = [0u8; 32]; + txid_b[0] = 2; + s.put_tx_full_batch_indexed(&[class_a_coinbase(txid_a, script.clone())], true) .unwrap(); - s.put_tx_full_batch_indexed(&[class_a_coinbase(txid_hi, low_script.clone())], true) + s.put_tx_full_batch_indexed(&[class_a_coinbase(txid_b, script.clone())], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([3u8; 32], vec![0x52])], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([4u8; 32], vec![0x53])], true) .unwrap(); - for shard in [0usize, 2, 3] { - let script = script_for_prefix_shard(shard, n_shards); - let mut txid = [0u8; 32]; - txid[0] = 10 + shard as u8; - s.put_tx_full_batch_indexed(&[class_a_coinbase(txid, script)], true) - .unwrap(); - } let udir = dir.join("unsorted"); - let out = crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); + rbitcoin_log::capture_logs(true); + let out = crate::collect_unsorted_shard_files(&s, &udir, n_shards, 2, None).unwrap(); + let logs = rbitcoin_log::take_logs(); + rbitcoin_log::capture_logs(false); + let info: Vec<&str> = logs + .iter() + .filter_map(|(level, m)| (*level == rbitcoin_log::Level::Info).then_some(m.as_str())) + .collect(); + assert!( + info.iter() + .any(|m| m.contains("scripthash keys collect start workers=") + && m.contains("budget_MiB=")), + "keys collect workers and budget, got {info:?}" + ); + let merge_starts: Vec<&str> = info + .iter() + .copied() + .filter(|m| m.contains("scripthash keys merge start")) + .collect(); + assert_eq!(merge_starts.len(), 1, "one keys merge start, got {info:?}"); + assert!( + merge_starts[0].contains("n_shards=") && merge_starts[0].contains("workers="), + "keys merge start workers, got {info:?}" + ); + assert!( + info.iter().any(|m| { + m.contains("scripthash keys merge shard=") + && m.contains("fold=") + && m.contains("bdz=") + }), + "live merge shard fold/bdz, got {info:?}" + ); assert_eq!(out.per_shard.len(), n_shards); assert!(unsorted_done_last_fk(&udir, n_shards).is_some()); - assert!(udir.join("DONE").is_file()); - assert_eq!(out.recs, 5); + assert!(udir.join("DONE.keys").is_file()); + let magic = std::fs::read(udir.join("DONE.keys")).unwrap(); + assert_eq!(&magic[0..8], b"SHKEYS02"); + assert!(!udir.join("DONE").is_file()); + let sh_dupe = script_hash(&script); + let si = prefix_shard_of(&sh_dupe, n_shards); + assert!( + !unsorted_keys_path(&udir, si).exists(), + "RAM merge unlinks keys/{si:02x}" + ); + let base = sorted_main_shard_path(s.path(), si, n_shards); + let h = MphfHead::open(&base).unwrap(); + let k_script = head_key_from_full(&sh_dupe); + assert_eq!(h.get(&k_script).unwrap().unwrap(), ShHeadValue::Empty); + assert_eq!( + h.get(&head_key_from_full(&script_hash(&[0x52]))) + .unwrap() + .unwrap(), + ShHeadValue::inline_one(Fk(3)) + ); + assert_eq!( + h.get(&head_key_from_full(&script_hash(&[0x53]))) + .unwrap() + .unwrap(), + ShHeadValue::inline_one(Fk(4)) + ); + assert_eq!(out.per_shard[si], 3, "dupe collapsed; two singles kept"); + assert!(unsorted_multi_fuse_path(&udir, si).is_file()); for shard in 0..n_shards { - let recs = decode_unsorted_file(&unsorted_shard_path(&udir, shard)); assert!( - recs.iter() - .all(|r| prefix_shard_of(&r.scripthash, n_shards) == shard), - "shard {shard} must contain only its prefix" + !unsorted_keys_path(&udir, shard).exists(), + "keys unlinked after head + fuse" ); - assert_eq!(recs.len() as u64, out.per_shard[shard]); } - let shard1 = decode_unsorted_file(&unsorted_shard_path(&udir, 1)); - assert_eq!(shard1.len(), 2); + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn keys_merge_seals_head_and_multi_fuse_without_keys_file() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + let n_shards = s.scripthash.head_shard_count(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([1u8; 32], vec![0x51])], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([2u8; 32], vec![0x51])], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([3u8; 32], vec![0x52])], true) + .unwrap(); + let udir = crate::unsorted_shard_dir(s.path()); + let out = crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); + let sh_dupe = script_hash(&[0x51]); + let si = prefix_shard_of(&sh_dupe, n_shards); + assert!( + !unsorted_keys_path(&udir, si).exists(), + "merge folds spills to head + fuse and unlinks keys/NN/" + ); + let base = sorted_main_shard_path(s.path(), si, n_shards); + assert!(MphfHead::exists(&base), "map fold writes scripthash.head"); + assert!( + unsorted_multi_fuse_path(&udir, si).is_file(), + "dupe mix64 lands in multi fuse8, not a unique keys file" + ); + assert_eq!(out.per_shard[si], 2, "dupe script + distinct script"); + let h = MphfHead::open(&base).unwrap(); + let k_dupe = head_key_from_full(&sh_dupe); + assert_eq!(h.get(&k_dupe).unwrap().unwrap(), ShHeadValue::Empty); + assert_eq!( + h.get(&head_key_from_full(&script_hash(&[0x52]))) + .unwrap() + .unwrap(), + ShHeadValue::inline_one(Fk(3)) + ); + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn leftover_shunsrt3_has_no_done_keys_and_recollects() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([1u8; 32], vec![0x51])], true) + .unwrap(); + let n_shards = s.scripthash.head_shard_count(); + let udir = crate::unsorted_shard_dir(s.path()); + std::fs::create_dir_all(&udir).unwrap(); + let mut buf = Vec::new(); + buf.extend_from_slice(b"SHUNSRT3"); + buf.extend_from_slice(&(n_shards as u32).to_le_bytes()); + buf.extend_from_slice(&1u64.to_le_bytes()); + for _ in 0..n_shards { + buf.extend_from_slice(&1u64.to_le_bytes()); + } + std::fs::write(udir.join("DONE"), &buf).unwrap(); + std::fs::write(unsorted_shard_path(&udir, 0), [0u8; 24]).unwrap(); + assert!(unsorted_done_last_fk(&udir, n_shards).is_none()); + crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); + assert!(udir.join("DONE.keys").is_file()); + assert!(!udir.join("DONE").is_file()); + assert!(!unsorted_shard_path(&udir, 0).is_file()); + assert!( + !unsorted_keys_path(&udir, 0).exists(), + "recollect merge writes head, not keys" + ); + let base = sorted_main_shard_path(s.path(), 0, n_shards); + assert!(MphfHead::exists(&base)); + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn mphf_from_keys_unlinks_key_file_and_hits() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([1u8; 32], vec![0x51])], true) + .unwrap(); + let n_shards = s.scripthash.head_shard_count(); + let udir = crate::unsorted_shard_dir(s.path()); + rbitcoin_log::capture_logs(true); + crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); + seal_mphf_from_keys(&s.scripthash, &udir, n_shards, None).unwrap(); + let logs = rbitcoin_log::take_logs(); + rbitcoin_log::capture_logs(false); + let sh = script_hash(&[0x51]); + let si = prefix_shard_of(&sh, n_shards); + let info: Vec<&str> = logs + .iter() + .filter_map(|(level, m)| (*level == rbitcoin_log::Level::Info).then_some(m.as_str())) + .collect(); + assert_eq!( + info.iter() + .filter(|m| m.contains("scripthash keys merge start")) + .count(), + 1, + "one keys merge start, got {info:?}" + ); + assert!( + info.iter().any(|m| { + m.contains(&format!("scripthash keys merge shard={si:02x}")) + && m.contains("fold=") + && m.contains("bdz=") + }), + "live merge shard fold/bdz, got {info:?}" + ); + assert!( + !unsorted_keys_path(&udir, si).exists(), + "keys dir unlinked after MPHF" + ); + let base = sorted_main_shard_path(s.path(), si, n_shards); + assert!(MphfHead::exists(&base)); + assert!(!MphfHead::exists(&unsorted_mphf_base(&udir, si))); + let h = MphfHead::open(&base).unwrap(); + let key = head_key_from_full(&sh); + assert!(h.slot_for_key16(&key).is_ok()); + let val = h.get(&key).unwrap().unwrap(); + assert_eq!(val, ShHeadValue::inline_one(Fk(1))); + assert!( + s.scripthash.unsealed_main_shards().contains(&si), + "head files are not a pack seal" + ); + assert!( + !unsorted_multi_fuse_path(&udir, si).is_file(), + "single key is not a fuse8" + ); + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn mphf_from_keys_multi_is_empty_and_existing_head_does_not_skip_pack() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([1u8; 32], vec![0x51])], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([2u8; 32], vec![0x51])], true) + .unwrap(); + let n_shards = s.scripthash.head_shard_count(); + let udir = crate::unsorted_shard_dir(s.path()); + crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); + seal_mphf_from_keys(&s.scripthash, &udir, n_shards, None).unwrap(); + let sh = script_hash(&[0x51]); + let si = prefix_shard_of(&sh, n_shards); + let base = sorted_main_shard_path(s.path(), si, n_shards); + let h = MphfHead::open(&base).unwrap(); + let key = head_key_from_full(&sh); + assert!(h.get(&key).unwrap().unwrap().is_empty()); + drop(h); + assert!( + unsorted_multi_fuse_path(&udir, si).is_file(), + "0xFFFF key writes unsorted/multi fuse8" + ); + let fuse = SealedFuse8::read_from(&unsorted_multi_fuse_path(&udir, si)).unwrap(); + assert!(fuse.contains(mix_key16(&key))); + assert!(s.scripthash.unsealed_main_shards().contains(&si)); + s.scripthash.reinit_empty_for_cold_materialize().unwrap(); + assert!( + s.scripthash.unsealed_main_shards().contains(&si), + "RAM unsealed after reinit even if head files exist" + ); + assert!(MphfHead::exists(&base)); + std::fs::remove_file(udir.join("DONE.keys")).unwrap(); + crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); + seal_mphf_from_keys(&s.scripthash, &udir, n_shards, None).unwrap(); + let h = MphfHead::open(&base).unwrap(); + assert!(h.get(&key).unwrap().unwrap().is_empty()); + assert!(s.scripthash.unsealed_main_shards().contains(&si)); + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn postings_collect_fuse_hits_skip_singles_and_logs_workers() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + let n_shards = s.scripthash.head_shard_count(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([1u8; 32], vec![0x51])], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([2u8; 32], vec![0x51])], true) + .unwrap(); + s.put_tx_full_batch_indexed(&[class_a_coinbase([3u8; 32], vec![0x52])], true) + .unwrap(); + let udir = crate::unsorted_shard_dir(s.path()); + crate::collect_unsorted_shard_files(&s, &udir, n_shards, 2, None).unwrap(); + seal_mphf_from_keys(&s.scripthash, &udir, n_shards, None).unwrap(); + rbitcoin_log::capture_logs(true); + crate::scripthash_materialize::collect_posts_covering( + &s.txs, + &s.scripthash, + &udir, + 2, + false, + None, + ) + .unwrap(); + let logs = rbitcoin_log::take_logs(); + rbitcoin_log::capture_logs(false); + let info: Vec<&str> = logs + .iter() + .filter_map(|(level, m)| (*level == rbitcoin_log::Level::Info).then_some(m.as_str())) + .collect(); + assert!( + info.iter() + .any(|m| m.contains("scripthash postings collect start workers=") + && m.contains("budget_MiB=")), + "postings workers and budget, got {info:?}" + ); + let k_multi = head_key_from_full(&script_hash(&[0x51])); + let k_single = head_key_from_full(&script_hash(&[0x52])); + let si = prefix_shard_of(&script_hash(&[0x51]), n_shards); + let posted = decode_post_keys(&udir, si); + assert!( + posted.contains(&k_multi), + "multi creates land in post, got {posted:?}" + ); + let n_single = posted.iter().filter(|k| **k == k_single).count(); assert!( - shard1[0].scripthash > shard1[1].scripthash, - "collect writes fk order, not scripthash order" + n_single <= 1, + "single is fuse-miss or at most an FP, got {n_single}" ); let _ = std::fs::remove_dir_all(&dir); } } +#[test] +fn pack_slot_stream_leaves_recs_empty_and_roundtrips() { + let dir = tmp(); + let t = four_shard_dir_table(&dir); + let n_shards = t.head_shard_count(); + let (low, high) = two_scripts_same_shard_reverse_hash(0, n_shards); + let sh_lo = script_hash(&low); + let sh_hi = script_hash(&high); + let k_lo = head_key_from_full(&sh_lo); + let k_hi = head_key_from_full(&sh_hi); + let base = sorted_main_shard_path(t.store_dir(), 0, n_shards); + MphfHead::write_pack8(&base, &[(k_lo, 0), (k_hi, 0)]).unwrap(); + let head = MphfHead::open(&base).unwrap(); + let slot_lo = head.slot_for_key16(&k_lo).unwrap(); + let slot_hi = head.slot_for_key16(&k_hi).unwrap(); + drop(head); + let mut session = t.pack_shard_session(0).unwrap(); + session.push_sorted_slot_fk(slot_lo, Fk(2)).unwrap(); + session.push_sorted_slot_fk(slot_lo, Fk(5)).unwrap(); + session.push_sorted_slot_fk(slot_hi, Fk(4)).unwrap(); + session.push_sorted_slot_fk(slot_hi, Fk(8)).unwrap(); + let pack = session.finish_pack().unwrap(); + assert!( + pack.recs.is_empty(), + "slot pack must not store a dummy key, recs={}", + pack.recs.len() + ); + assert_eq!(pack.slot_words.len(), 2); + assert!(pack.slot_words.iter().any(|(s, _)| *s == slot_lo)); + assert!(pack.slot_words.iter().any(|(s, _)| *s == slot_hi)); + t.publish_packed_shard(0, pack).unwrap(); + let fks = |sh: &[u8; 32]| -> Vec { + t.entries(sh).unwrap().into_iter().map(|e| e.0 .0).collect() + }; + assert_eq!(fks(&sh_lo), vec![2, 5]); + assert_eq!(fks(&sh_hi), vec![4, 8]); + let _ = std::fs::remove_dir_all(&dir); +} + #[test] fn unsorted_pack_sorts_numeric_fk_and_keeps_all_creates() { { @@ -2250,27 +2564,22 @@ fn unsorted_pack_sorts_numeric_fk_and_keeps_all_creates() { let table = four_shard_dir_table(&sh_dir); let n_shards = 4usize; let udir = sh_dir.join(UNSORTED_SHARD_DIR); - std::fs::create_dir_all(&udir).unwrap(); + std::fs::create_dir_all(udir.join("keys")).unwrap(); + std::fs::create_dir_all(udir.join("post")).unwrap(); let (low, high) = two_scripts_same_shard_reverse_hash(1, n_shards); let sh_lo = script_hash(&low); let sh_hi = script_hash(&high); - let mut bytes = Vec::new(); - let mut push = |sh: [u8; 32], fk: u64| { - let mut r = [0u8; 24]; - r[..16].copy_from_slice(&sh[..16]); - r[16..].copy_from_slice(&fk.to_le_bytes()); - bytes.extend_from_slice(&r); - }; - push(sh_hi, 256); - push(sh_hi, 2); - push(sh_lo, 3); - push(sh_lo, 1); - push(sh_lo, 1); - push(sh_hi, 0); - std::fs::write(unsorted_shard_path(&udir, 1), &bytes).unwrap(); + let k_lo = head_key_from_full(&sh_lo); + let k_hi = head_key_from_full(&sh_hi); + write_keys_spill_entries(&udir, 1, 0, &[(k_lo, None), (k_hi, None)]).unwrap(); for shard in [0usize, 2, 3] { - std::fs::write(unsorted_shard_path(&udir, shard), []).unwrap(); + write_keys_spill_entries(&udir, shard, 0, &[]).unwrap(); } + seal_mphf_from_keys(&table, &udir, n_shards, None).unwrap(); + drop(MphfHead::open(sorted_main_shard_path(&sh_dir, 1, n_shards)).unwrap()); + write_post_spill_entries(&udir, 1, 0, &[(k_hi, &[2u64][..]), (k_lo, &[3u64, 1][..])]) + .unwrap(); + write_post_spill_entries(&udir, 1, 1, &[(k_hi, &[256u64][..])]).unwrap(); rbitcoin_log::capture_logs(true); let mat = materialize_sh_from_unsorted(&table, &udir, 1, None).unwrap(); let logs = rbitcoin_log::take_logs(); @@ -2286,19 +2595,6 @@ fn unsorted_pack_sorts_numeric_fk_and_keeps_all_creates() { }) .collect(); assert_eq!(done.len(), 4, "one finish line per shard, got {logs:?}"); - for shard in 0..n_shards { - let tag = format!("shard={shard:02x}"); - assert!( - done.iter() - .any(|m| m.contains(&tag) && m.contains("elapsed=")), - "missing finish log for {tag}: {done:?}" - ); - } - assert!( - done.iter() - .any(|m| m.contains("shard=01") && m.contains("keys=2") && m.contains("creates=4")), - "data shard must log packed keys/creates: {done:?}" - ); let mut lo: Vec = table .entries(&sh_lo) .unwrap() @@ -2317,12 +2613,100 @@ fn unsorted_pack_sorts_numeric_fk_and_keeps_all_creates() { assert_eq!( hi, vec![2, 256], - "fk 256 must sort after 2, not as LE bytes" + "fk 256 must sort after 2 after reconstruct from two segments" ); let _ = std::fs::remove_dir_all(&dir); } } +#[test] +fn unsorted_pack_skips_one_fk_fuse_fp_and_rewrites_multi() { + { + let dir = tmp(); + let sh_dir = dir.join("sh4"); + std::fs::create_dir_all(&sh_dir).unwrap(); + let table = four_shard_dir_table(&sh_dir); + let n_shards = 4usize; + let udir = sh_dir.join(UNSORTED_SHARD_DIR); + std::fs::create_dir_all(udir.join("keys")).unwrap(); + std::fs::create_dir_all(udir.join("post")).unwrap(); + let script_fp = script_for_prefix_shard(1, n_shards); + let script_multi = script_for_prefix_shard(2, n_shards); + let sh_fp = script_hash(&script_fp); + let sh_multi = script_hash(&script_multi); + let k_fp = head_key_from_full(&sh_fp); + let k_multi = head_key_from_full(&sh_multi); + write_keys_spill_entries(&udir, 1, 0, &[(k_fp, Some(Fk(7)))]).unwrap(); + write_keys_spill_entries(&udir, 2, 0, &[(k_multi, None)]).unwrap(); + for shard in [0usize, 3] { + write_keys_spill_entries(&udir, shard, 0, &[]).unwrap(); + } + seal_mphf_from_keys(&table, &udir, n_shards, None).unwrap(); + let bump_fp = table.bodies[1].logical_len(); + write_post_spill_entries(&udir, 1, 0, &[(k_fp, &[7u64][..])]).unwrap(); + write_post_spill_entries(&udir, 2, 0, &[(k_multi, &[10u64, 11][..])]).unwrap(); + rbitcoin_log::capture_logs(true); + let mat = materialize_sh_from_unsorted(&table, &udir, 2, None).unwrap(); + let logs = rbitcoin_log::take_logs(); + rbitcoin_log::capture_logs(false); + let info: Vec<&str> = logs + .iter() + .filter_map(|(level, m)| (*level == rbitcoin_log::Level::Info).then_some(m.as_str())) + .collect(); + assert!( + info.iter() + .any(|m| m.contains("scripthash unsorted pack start") && m.contains("workers=2")), + "pack start workers, got {info:?}" + ); + if let Some(pack_done_idx) = info + .iter() + .position(|m| m.contains("scripthash unsorted pack done")) + { + if let Some(first_shard_idx) = info + .iter() + .position(|m| m.contains("scripthash unsorted pack shard=")) + { + assert!( + first_shard_idx < pack_done_idx, + "pack shard line before pack done, got {info:?}" + ); + } + } + assert_eq!( + table.unsealed_main_shards().len(), + 0, + "pack_workers=2 must finish every unsealed shard" + ); + assert_eq!( + table.bodies[1].logical_len(), + bump_fp, + "1-fk skip must not grow body" + ); + assert_eq!( + table + .entries(&sh_fp) + .unwrap() + .into_iter() + .map(|e| e.0 .0) + .collect::>(), + vec![7] + ); + let mut multi: Vec = table + .entries(&sh_multi) + .unwrap() + .into_iter() + .map(|e| e.0 .0) + .collect(); + multi.sort_unstable(); + assert_eq!(multi, vec![10, 11]); + let h = MphfHead::open(sorted_main_shard_path(&sh_dir, 2, n_shards)).unwrap(); + let val = h.get(&k_multi).unwrap().unwrap(); + assert!(!val.is_empty(), "2+ fks rewrite Empty → body"); + assert!(mat.creates >= 2); + let _ = std::fs::remove_dir_all(&dir); + } +} + #[test] fn unsorted_combined_skips_collect_when_done_and_resumes_unsealed() { { @@ -2342,8 +2726,11 @@ fn unsorted_combined_skips_collect_when_done_and_resumes_unsealed() { std::fs::create_dir_all(&sh_dir).unwrap(); let table = four_shard_dir_table(&sh_dir); let udir = sh_dir.join(UNSORTED_SHARD_DIR); - crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); - materialize_sh_from_unsorted(&table, &udir, 1, None).unwrap(); + crate::scripthash_materialize::collect_unsorted_covering_txs( + &s.txs, &table, &udir, n_shards, 1, false, None, + ) + .unwrap(); + materialize_sh_from_unsorted_from_txs(&table, &s.txs, &udir, 1, 1, None).unwrap(); assert!(table.unsealed_main_shards().is_empty()); let again = materialize_sh_from_unsorted(&table, &udir, 2, None).unwrap(); assert_eq!(again.creates, 4); @@ -2370,9 +2757,9 @@ fn unsorted_cancel_before_collect_is_cancelled() { let udir = crate::unsorted_shard_dir(s.path()); assert!( unsorted_done_last_fk(&udir, s.scripthash.head_shard_count()).is_none(), - "cancel must not write DONE" + "cancel must not write DONE.keys" ); - assert!(!udir.join("DONE").is_file()); + assert!(!udir.join("DONE.keys").is_file()); let _ = std::fs::remove_dir_all(&dir); } } @@ -2387,7 +2774,7 @@ fn unsorted_done_records_class_a_last_fk() { let n_shards = s.scripthash.head_shard_count(); let udir = crate::unsorted_shard_dir(s.path()); let out = crate::collect_unsorted_shard_files(&s, &udir, n_shards, 1, None).unwrap(); - assert!(udir.join("DONE").is_file()); + assert!(udir.join("DONE.keys").is_file()); assert_eq!(out.last_fk, s.txs.count()); assert_eq!( crate::unsorted_done_last_fk(&udir, n_shards), @@ -2420,8 +2807,87 @@ fn unsorted_materialize_appends_when_done_lags_and_no_shards() { assert_eq!( s.scripthash.entries(&script_hash(&[0x52])).unwrap().len(), 1, - "Class A grown after DONE must be appended into unsorted before pack" + "Class A grown after DONE.keys must be recollected before pack" + ); + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn pack_shard_unlinks_only_that_post_file() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + let n_shards = 4usize; + for shard in 0..n_shards { + let script = script_for_prefix_shard(shard, n_shards); + let mut txid = [0u8; 32]; + txid[0] = shard as u8; + s.put_tx_full_batch_indexed(&[class_a_coinbase(txid, script)], true) + .unwrap(); + } + let sh_dir = dir.join("sh4"); + std::fs::create_dir_all(&sh_dir).unwrap(); + let table = four_shard_dir_table(&sh_dir); + let udir = sh_dir.join(UNSORTED_SHARD_DIR); + collect_unsorted_covering_txs(&s.txs, &table, &udir, n_shards, 1, false, None).unwrap(); + seal_mphf_from_keys(&table, &udir, n_shards, None).unwrap(); + materialize_sh_from_unsorted_from_txs(&table, &s.txs, &udir, 1, 1, None).unwrap(); + assert!( + table.unsealed_main_shards().is_empty(), + "second pack via from_txs finishes remaining shards" ); + for shard in 0..n_shards { + assert!( + !unsorted_post_path(&udir, shard).exists(), + "post/{shard:02x} unlinked after seal" + ); + } + let _ = std::fs::remove_dir_all(&dir); + } +} + +#[test] +fn pack_one_shard_unlinks_only_its_postings() { + { + let dir = tmp(); + let s = crate::Store::create_tiny(&dir).unwrap(); + let n_shards = 4usize; + let mut keys = Vec::new(); + for shard in 0..n_shards { + let script = script_for_prefix_shard(shard, n_shards); + keys.push(script_hash(&script)); + let mut txid = [0u8; 32]; + txid[0] = shard as u8; + s.put_tx_full_batch_indexed(&[class_a_coinbase(txid, script)], true) + .unwrap(); + } + let sh_dir = dir.join("sh4"); + std::fs::create_dir_all(&sh_dir).unwrap(); + let table = four_shard_dir_table(&sh_dir); + let udir = sh_dir.join(UNSORTED_SHARD_DIR); + collect_unsorted_covering_txs(&s.txs, &table, &udir, n_shards, 1, false, None).unwrap(); + seal_mphf_from_keys(&table, &udir, n_shards, None).unwrap(); + crate::scripthash_materialize::collect_posts_covering( + &s.txs, &table, &udir, 1, false, None, + ) + .unwrap(); + pack_one_extract_shard(&table, &udir, 0).unwrap(); + assert!(!unsorted_post_path(&udir, 0).exists(), "post/00 unlinked"); + let unsealed = table.unsealed_main_shards(); + assert!(!unsealed.contains(&0), "shard 0 packed"); + for shard in 1..n_shards { + assert!( + unsealed.contains(&shard), + "shard {shard} remains unsealed until pack, unsealed={unsealed:?}" + ); + } + assert_eq!(table.entries(&keys[0]).unwrap().len(), 1); + materialize_sh_from_unsorted(&table, &udir, 1, None).unwrap(); + assert!(table.unsealed_main_shards().is_empty()); + for k in &keys { + assert_eq!(table.entries(k).unwrap().len(), 1); + } let _ = std::fs::remove_dir_all(&dir); } } diff --git a/crates/rbitcoin-store/src/tx_table/mod.rs b/crates/rbitcoin-store/src/tx_table/mod.rs index 21c89d592..6ec6129e2 100644 --- a/crates/rbitcoin-store/src/tx_table/mod.rs +++ b/crates/rbitcoin-store/src/tx_table/mod.rs @@ -11,7 +11,7 @@ use rbitcoin_primitives::{read_uleb128, write_uleb128, Fk, TableKind}; use std::path::Path; use std::sync::Arc; -/// Host RAM budget per parallel `tx.head` rebuild worker (not SH pack's 2 GiB). +/// Host RAM budget per parallel `tx.head` rebuild worker (not SH extract's 1.5 GiB). /// BDZ peel scratch + keys + g at the default 2²⁵ seal is ≈1 GiB peak. pub const TX_HEAD_REBUILD_WORKER_FREE_RAM_BYTES: u64 = 1024 * 1024 * 1024; @@ -2054,7 +2054,7 @@ impl TxTable { /// Range width is [`Self::rebuild_seal_keys`] (default 2²⁵). Remainder is /// sealed too; an empty open tail is created for later inserts. /// Workers: [`Self::rebuild_workers`] (min of CPUs, free RAM / 1 GiB, - /// and range count). Distinct from SH pack's 2 GiB cap. + /// and range count). Distinct from SH extract's 1.5 GiB cap. pub fn rebuild_head_from_bodies( &self, mut on_progress: impl FnMut(u64, u64, u64), diff --git a/docs/concurrency.md b/docs/concurrency.md index 6da138c3f..1dbf7f297 100644 --- a/docs/concurrency.md +++ b/docs/concurrency.md @@ -69,9 +69,9 @@ Do not enter Tip until IBD catch-up complete: no best-chain remainder on-path getdata) and path high water at or within 1 of max peer height (one-block version chatter only when `headers_done`). Competing `hash_height` and leftover explore getdata are not remainder. Tip entry -bulk-materializes SH (Class A collect → unsorted per-shard files → -in-place unique-sort + seal `scripthash.head/NN`; pack workers capped at -one per 2 GiB host free RAM). Shared file `scripthash.body` +bulk-materializes SH (two Class A `txout` scans → identity-map key spills +then fuse-hit postings → BDZ `scripthash.head/NN`; extract workers capped at +one per 1.5 GiB host free RAM). Shared file `scripthash.body` is one writer. Overflow body is one writer (ingest / compact). It does **not** rebuild `tx.head` or spend annotations. diff --git a/docs/crash-recovery.md b/docs/crash-recovery.md index 7a13a86cd..2121d704b 100644 --- a/docs/crash-recovery.md +++ b/docs/crash-recovery.md @@ -100,9 +100,9 @@ Private, **not** Class A. RAM graph is source of truth; files may lag. - Direct IBD keeps segmented **`tx.head/`** (archive) and **spend annotations** (confirm) live; tip entry does **not** re-scan Class A to repair them. Corrupt head/spends ⇒ reindex (optional manual `backfill_tx_index` rebuilds segmented head mappings from Class A). - **Segmented `tx.head`:** directory `tx.head/` with `meta` + open OA `NNNNNN`; sealed `NNNNNN.mphf` + `.fuse8`. Packed BDZ `g` is FdOnly (4 KiB page stream); MPHF output is `rel−1`. Flat `tx.head.meta` / `tx.head.NNNNNN` are **migrated into** `tx.head/` on open. Roll opens the next OA first; seal runs on a sidecar and publishes later. Seal/install of `meta` / `.mphf` is sibling tmp + `sync_all` then rename (empty truncate of the live name is not a seal). Publish persists sealed `meta` **before** unlinking the segment's OA; a leftover OA next to a sealed `.mphf` is discarded on open. Kill mid-seal leaves **at most one** unsealed non-tail OA: open collects fuse keys once from `txid.body` and seals it (does not retain `open_keys`). Two unsealed non-tails is **Corrupt**. Leftover fuse8 v1 and flat `tx.head.meta` **refuse** (`Query::open`); wipe `store/tx.head` (Class A kept) then restart. Unreadable **current** head (empty/truncated `meta`, truncated/missing sealed MPHF) with Class A: wipe+rebuild from `txid.body` (same cost as a clean wipe). Wipe or empty occupancy + Class A: open rebuilds **MPHF+fuse8 directly** from `txid.body` in parallel (default **2²⁵** keys/range; `RBITCOIN_TX_HEAD_REBUILD_WORKERS`); no historical OA. Legacy mono `tx.head` file / `.new` / `.resize` are not opened — reindex. -- Scripthash: Direct IBD **defers** SH (no memtable, no confirm enqueue). After the horizon, one Class A pass writes unsorted `scripthash.unsorted/NN`, then unique-sort + seal. A **durable head** on restart stays Tip: write-behind / `recover_sh_writebehind` fills any HWM lag; leftover `scripthash.runs` are discarded (not WarmOnly-merged). - - **Full cold** when head empty. **`RBITCOIN_SH_FORCE_REBUILD=1`:** wipe head + full Class A collect + unsorted pack. Empty collect after Class A creates remain is fatal. - - **Cold resume:** sealed `scripthash.head/NN` is the commit (holes stay). Incomplete collect (no `DONE`) restarts the Class A pass. `DONE` records the inclusive Class A `create_fk` scanned (`SHUNSRT3`). If Class A grew after `DONE` and **no** shards are sealed, collect **appends** `(last_fk, tip]` into the unsorted files. If any shard is already sealed, pack remaining unsealed files as-is, then Class A tail-append onto the durable head (Direct write-behind no-ops). Leftover `SHUNSRT2` (no last_fk) restarts collect. After all shards seal, the unsorted dir is removed. +- Scripthash: Direct IBD **defers** SH (no memtable, no confirm enqueue). After the horizon, two Class A `txout` scans: each worker owns a contiguous create-fk span and unsized maps (1.5 GiB estimate cap; spill the largest shard map while over budget; one writer, 1-slot queue) into `scripthash.unsorted/keys/NN/` (`SHKSP01` files, first-fk delta singles; tmp+rename, not a kill-9 barrier — no `DONE.keys` still wipes unsorted); merge folds those spills into one map, one walk to `scripthash.head/NN` + `multi/NN.fuse8`, and unlinks `keys/NN/`; then fuse-hit `post` (`SHPST01` spills under `post/NN/`), pack folds those spills then 2+ bodies, unlink each shard's extract as it seals. A **durable head** on restart stays Tip: write-behind / `recover_sh_writebehind` fills any HWM lag; leftover `scripthash.runs` are discarded (not WarmOnly-merged). + - **Full cold** when head empty. **`RBITCOIN_SH_FORCE_REBUILD=1`:** wipe head + full two-scan collect + pack. Empty collect after Class A creates remain is fatal. + - **Cold resume:** RAM-published `scripthash.head/NN` is the pack commit (holes stay). `MphfHead::exists` after pass-1 BDZ is **not** pack-done. No valid `DONE.keys` (including leftover `SHUNSRT3` / 24 B `NN`, or `keys/NN` / `post/NN` as a file) deletes unsorted and restarts pass 1. A spill whose first 8 B are not `SHKSP01` is Corrupt — wipe unsorted; do not parse or migrate. `DONE.keys` / `DONE.post` record the inclusive Class A `create_fk` scanned (`SHKEYS02` last_fk marker / `SHPOST02`). Missing current `keys/NN/` spills with unsealed shards finishes BDZ from those files. Head files present and no `DONE.post` restarts pass 2 from fk 1. A `post/NN` **file**, or a spill whose first 8 B are not `SHPST01`, is Corrupt (wipe unsorted). If Class A grew after `DONE.post` and **no** shards are packed, append postings. If any shard is already RAM-sealed, pack remaining unsealed `post/NN/`, then Class A tail-append onto the durable head (Direct write-behind no-ops). After all shards seal, the unsorted dir is removed. Empty head + leftover catalog: wipe leftover runs + SEAL, then Class A collect (not k-way from `scripthash.runs`). Durable head: leftover runs are discarded, `SEAL` kept; missing `include_hwm` bootstraps from SEAL. Inclusion HWM: `scripthash.include_hwm`. **Leftover live OA** at `scripthash.head` (or non-`SHSR` `ovf/NNNNNN`): refuse — wipe `store/scripthash*` and restart with `--shindex`. **SH head open:** sealed **main** shards load `.idx` only (one entry per 128 records; no fuse). Sealed **ovf** loads `.idx` + BF8R. Occupancy scan is not diff --git a/docs/env-knobs.md b/docs/env-knobs.md index 7e1e63c02..62dc89ab2 100644 --- a/docs/env-knobs.md +++ b/docs/env-knobs.md @@ -27,11 +27,11 @@ for signet/mainnet sync. **Not** CLI. | `RBITCOIN_CLASS_C_INRAM_MAX_MB` | 256 | L2 cap for `confirmed` / `header_txs_*`; over → fd L0. `strong_tx` always L2 | | `RBITCOIN_TX_HEAD_BITS` | scale default | `tx.head` bits (dangerous on a live datadir) | | `RBITCOIN_TX_HEAD_REBUILD_SEAL_BITS` | 25 | Wipe/empty-head MPHF range `2^bits` (26 wider; clamp 6..=26) | -| `RBITCOIN_TX_HEAD_REBUILD_WORKERS` | min(n-cpu, free-RAM/1 GiB) | Wipe/empty-head MPHF parallelism (`1` = serial). Unset = auto. **Not** SH pack's 2 GiB cap | +| `RBITCOIN_TX_HEAD_REBUILD_WORKERS` | min(n-cpu, free-RAM/1 GiB) | Wipe/empty-head MPHF parallelism (`1` = serial). Unset = auto. **Not** the SH extract 1.5 GiB cap | | `RBITCOIN_HEAD_SLOTS_HEADER` | scale default | Header hash-head initial slots (power of two) | | `RBITCOIN_SH_UNIQUE_HINT` | off | SH unique-hint probe | | `RBITCOIN_SH_FORCE_REBUILD` | off | Sticky SH rebuild (also in OPERATOR) | -| `RBITCOIN_SH_MERGE_WORKERS` | min(n-cpu, free-RAM/2 GiB) | Unsorted SH pack (`1` = serial). Unset = auto (see [`ibd-memory.md`](./ibd-memory.md)) | +| `RBITCOIN_SH_MERGE_WORKERS` | min(n-cpu, max(1, free-RAM/1.5 GiB)) | SH extract (collect, merge, BDZ, fuse, pass 2, pack). `1` = serial. Unset = auto (see [`ibd-memory.md`](./ibd-memory.md)) | | `RBITCOIN_P2P_MAX_INBOUND` | 125 | Only if `--max-inbound` / conf omitted | | `RBITCOIN_URING_DRAIN_HARD_SECS` | 120 | `drain_all` zero-CQE abort (Linux/pool/IOCP). Raise if completions are slow but still arriving | | `RBITCOIN_RPC_WAIT_TIP_IDLE` | off | `getblockcount` / `getbestblockhash` / `wait_height` wait until the tip-accept lane is empty. Default (unset) waits only for the job that was running when the RPC arrived. The Core-functional bitcoind shim sets `1` so `sync_blocks` stays deterministic. **Not** a production operator knob | diff --git a/docs/heads.md b/docs/heads.md index 2aeb3ee20..173251e13 100644 --- a/docs/heads.md +++ b/docs/heads.md @@ -29,7 +29,7 @@ spk hash ──► scripthash.head/NN.mphf+.val sealed BDZ3 main (2-bit g + |--------|---------|-------------|--------------| | `HashHead` | `header.head` (+ `.gN`) | header **hash prefix** → header fk (`.mlt` if several) | Header ensure / `has_block` / prev walk | | `AddressHead` + `SegmentedTxHead` | `tx.head/` (`meta`, open `NNNNNN`, sealed `.mphf|.fuse8`) | **mixed txid** → relative **create_fk** (body-verify on `txid.body`). Live OA rolls at 80% slots; wipe-rebuild seals **2²⁵** keys/range in parallel (no OA). Lookup maps `.fuse8` read-only (heap `fuse8=0`); packed BDZ `g` is FdOnly (4 KiB page stream); MPHF output is `rel−1`. Open OA is keyless; the seal sidecar collects keys from `txid.body`. | Confirm **lookup** stamp after live pin miss | -| Sealed SH main | `scripthash.head/NN.mphf` + `.val` | Electrum **scripthash prefix** → pack8 locators. Compact BDZ3: packed 2-bit `g` FdOnly; occupancy mapped (prefix through occ, not tags); tags/val FdOnly. | After tip bulk | +| Sealed SH main | `scripthash.head/NN.mphf` + `.val` | Electrum **scripthash prefix** → pack8 locators. Compact BDZ3: packed 2-bit `g` FdOnly; occupancy mapped (prefix through occ, not tags); tags/val FdOnly. Pass-1 extract BDZ writes this path (`inline_one` vs Empty) before pack; **`MphfHead::exists` is not pack-done** (RAM unsealed until `publish_packed_shard`). | After tip bulk | | Ingest + L0/L1 ovf | `scripthash.ovf/ingest`, L0 `SHSR`, L1 MPHF, `ovf/body` | Same pack8 key for incremental / post-seal new keys | Tip; lookup ingest → L0 → L1 → main | `tx.head` is **not** a `HashHead`. Header slots come from `HeadScale` (`RBITCOIN_HEAD_SLOTS_HEADER` override). Sorted/MPHF SH shards are `sh_main_shard_count` (tiny=1, mainnet=64), not a HashHead. Leftover 256-way `header.head/` is Layout refuse. diff --git a/docs/ibd-memory.md b/docs/ibd-memory.md index a38ccc781..166fd83bb 100644 --- a/docs/ibd-memory.md +++ b/docs/ibd-memory.md @@ -43,8 +43,8 @@ pres and **not** the raw bytes. Reorg gather that wants wire re-encodes. | **ConfirmParentCache header plans** | tip-GC window | Always on — required for multi-block wire MTP | | **Confirm plans / headers** | offer-ahead window | `ConfirmParentCache::advance_tip` from write `post_commit` | | **SH catalog runs** | leftover `scripthash.runs` discarded at tip (unsorted collect does not write them) | write-behind / discard; not during Direct confirm | -| **SH unsorted collect / pack** | Collect: nCPU (no env / RAM cap; 1 MiB grow-on-demand write buffers; per-shard mutex so pwrite issues in offset order; 64 MiB fallocate on full flushes). Pack: min(CPUs, free RAM / 2 GiB); `RBITCOIN_SH_MERGE_WORKERS` override. One anonymous file image per pack worker (in-place unique-sort; MPHF has no HashSet). Class A collect spans 1 MiB | Tip finalize. Unsorted files under `scripthash.unsorted/` | -| **`tx.head` wipe-rebuild workers** | min(CPUs, host free RAM / 1 GiB, range count); floor 1. Same free-RAM probe as SH. Matches BDZ peel+keys+g peak at default 2²⁵. **Not** the SH pack 2 GiB cap. Env `RBITCOIN_TX_HEAD_REBUILD_WORKERS` override (`1` = serial) | Empty/wipe `tx.head` rebuild from `txid.body`. Logs `workers=` `free_GiB=` | +| **SH unsorted collect / pack** | Two `txout` scans. All extract phases share `sh_extract_workers()` = min(CPUs, max(1, free RAM / 1.5 GiB)); `RBITCOIN_SH_MERGE_WORKERS` override. **Collect maps** use that same 1.5 GiB **per worker** as a cap (not a prefault): each worker owns a contiguous create-fk span and `n_shards` unsized identity maps (`HashMap`, `0` = multi; 64 B/key estimate). After each 64 k-fk loc/body batch, spill the largest shard map while `sum(estimate) >= 1.5 GiB`. Status `scanned=` is finished fks (not max fk). Spills, including the end flush, go through one writer and a 1-slot queue so encode+tmp+rename is serial. Fuse8 mmap on pass 2 is process-wide, not subtracted from the cap. `keys/NN/` is a **directory** of `SHKSP01` spill files (`000000`, …; n_multi key16s already `0`, then first-fk-sorted uleb(delta)‖key16 singles; tmp+rename, no `sync_all`). Merge folds those spill files only into one identity map, one walk to `scripthash.head/NN` pack8 and `multi/NN.fuse8` (file, not kept resident for other shards), unlinks `keys/NN/`. The folded map is consumed into the pack8 records and dropped before fuse8 and BDZ. A previous `DONE` / 24 B `NN` layout, or `keys/NN` as a file, with no valid `DONE.keys` is deleted and pass 1 restarts. A spill whose magic is not `SHKSP01` is Corrupt. Pass 2 keeps fuse8 mmap only (no BDZ). Same static fk spans; fuse-hit creates append `fk` to per-worker `HashMap>` (skip `vec.last() == fk`). Estimate `80 × n_keys + 8 × n_fks`; spill-largest while over 1.5 GiB. `post/NN/` is a **directory** of `SHPST01` spill files (`000000`, …; `u32 n_keys` then `key16 ‖ uleb(n) ‖ n × uleb(fk − prev)`; tmp+rename, no `sync_all`). A `post/NN` file, or a spill whose magic is not `SHPST01`, is Corrupt. **Pack fold** (named, after collect maps drop): one shard ~1.2 M multi keys × ~30 fks ≈ 0.3–0.5 GiB; 8 pack workers a few GiB. Fold spills into one map, then `MphfHead::open` + `slot_for_key16` + 2+ body + `rewrite_val_slots`; `len == 1` after fold is `fp_singles` (pass-1 `inline_one` stays). Unlink `post/NN/`. Packed 2-bit `g` ~8 MiB per pack worker + 16 MiB `body_buf`. Inner loc/body batch 65 536 fks inside each worker span (not a steal grain); body IO 16 MiB | Tip finalize. Extract under `scripthash.unsorted/{keys,multi,post}` | +| **`tx.head` wipe-rebuild workers** | min(CPUs, host free RAM / 1 GiB, range count); floor 1. Same free-RAM probe as SH. Matches BDZ peel+keys+g peak at default 2²⁵. **Not** the SH extract 1.5 GiB cap. Env `RBITCOIN_TX_HEAD_REBUILD_WORKERS` override (`1` = serial) | Empty/wipe `tx.head` rebuild from `txid.body`. Logs `workers=` `free_GiB=` | | **Ordered work path** | `MAX_ORDERED_HEADERS` | `IbdWorkState::hygiene` | Tests that need a clean process must call these **same** entry points (or drop the @@ -181,7 +181,7 @@ touches. Census: [`SCHEMA.md`](../SCHEMA.md) (tip 962298, 1.42 B creates). | **Comfortable serve** (busy wallets, Electrum tweaks, RPC reconstruct) | Above + more `txout` + SH body slabs + `txid.body` | **16–32 GiB** | `inwit` except rawtx | | **IBD pin+annotate (no thrash)** | **All** `txout` + **all** `spent` + `create.loc` (~3 GiB) + `txid.body` + `tx.head` | **~221 GiB** | **`inwit` (~486 GiB)** — wire still holds witness | | **IBD + reconstruct/getdata** | Previous + `inwit` | **~710 GiB** (same order as old packed `tx.body`) | — | -| **SH tip materialize** | Unsorted shards: nCPU Class A collect (16 MiB libc `pread` `txout` spans + 1 MiB write buffers); pack workers min(CPUs, free RAM / 2 GiB); ingest OA **~768 MiB** (2²⁵×24 B) | **~2 GiB** heap per pack worker | No catalog k-way pages | +| **SH tip materialize** | Two `txout` scans (16 MiB libc `pread` spans + 1 MiB write buffers); extract workers min(CPUs, free RAM / 1.5 GiB); ingest OA **~768 MiB** (2²⁵×24 B) | **~1.5 GiB** heap per extract worker (BDZ `g` + fuse + maps); pack ~8 MiB packed `g` + 16 MiB `body_buf` | No catalog k-way pages; no 64 × `.val` random-read set on the scan | Packed schema 13/14 needed the whole **`tx.body` (~663 GiB)** hot for the same pin/annotate work. Split Class A drops that to **~161 GiB** (`txout`+`spent`) diff --git a/docs/io-modality.md b/docs/io-modality.md index ca82f2d79..ec68399a5 100644 --- a/docs/io-modality.md +++ b/docs/io-modality.md @@ -142,7 +142,7 @@ RAM fence (~15 MiB at 1M blocks), not a file. | Spend pure-write annotate | `RBITCOIN_IO` | uring/pwrite or pwrite on **`spent.body` FD** | | Class C create-height | (RAM fence) | no IO | | Class A body/idx **linear append** | always | **pwrite** (three stems + three idx) | -| SH unsorted collect (Class A) | libc `pread` | Sequential coalesced **16 MiB** `txout.body` spans (nCPU workers). Not TLS uring: one large positional read per span, not a completion machine. Writes are libc `pwrite`. | +| SH unsorted collect (Class A) | libc `pread` | **Two** sequential coalesced **16 MiB** `txout.body` scans (`sh_extract_workers`): static create-fk spans (no steal); per-worker identity maps spill-largest as `SHKSP01` under `keys/NN/` (one spill writer, 1-slot queue; tmp+rename, no `sync_all`; first-fk delta singles); merge folds those files into one map, one walk to `scripthash.head/NN` + `multi/NN.fuse8`; leftover keys shapes refuse. Then fuse-hit `SHPST01` post spills under `post/NN/` (`80n+8f` estimate, same 1.5 GiB worker cap and 1-slot writer; tmp+rename, no `sync_all`). Not TLS uring: one large positional read per span, not a completion machine. Writes are libc `pwrite` / tmp+rename. Pass 2 does not Fd-index the main MPHF; pack folds one shard's spills then `slot_for_key16` once per unique multi key. Packed 2-bit `g` is ~8 MiB per pack worker, not a 12 GiB `.val` page-cache set. | | SH Electrum/Esplora join | `RBITCOIN_IO` | Query-thread session: waved `idx_body_pipeline` on **`txout.body`**, optional page-grouped **`txid.body`** (history: creates+spenders; listunspent: unspent creates; balance/`/address` stats: none), `get_spender_meta_at_abs_batch` on **`spent.body`**. Megakey **extent**: one span pread of `extent_n` pages, then linked 4 KiB tail. Schema-18 mode 10 leftovers **refuse** on open. Does **not** share a confirm TLS ring. | | SP-tweak backfill | `RBITCOIN_IO` | idx ranges **before** TLS ring; uring/pread `txout.body`, then `inwit` + parent `txout` for P2TR only. Writes are **not** the load ring: batched `sp_tweaks` body+idx pwrite (not per-tx CQEs). | diff --git a/docs/quality.md b/docs/quality.md index abb990c3f..bb3a6a244 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -34,9 +34,10 @@ Counts and procedures stay in the owner doc. A row is the rank and the outcome. | 4 | **R-10** | Residual god-files | Peel **only** when a higher row needs a seam. Do not split `interpreter.rs` opcode `match` or io_uring machines. Named extracts: **Q-61** Completed. **Q-56** Completed (scriptnum + pack-ints). **Q-68** Completed (create.loc SIMD). | | 5 | **Q-67** | `asked_blocks` clone on hold | `hold_body` clones `asked_blocks` before `held_bodies` insert so the read lock does not overlap the write (`HeldBodies::insert` already takes `&HashSet`). Bound is `MAX_SERVE_BLOCKS` × peers. Follow-up: pass the read guard with a documented lock order, or keep the clone as a named trade. Owner: `crates/rbitcoin-net/src/chain.rs`. | | 6 | **Q-69** | CLN / LDK chain backend | Operator can point CLN `bcli` and ldk-node Esplora/Electrum at this node. Owner: [`lightning.md`](./lightning.md). | +| 7 | **Q-70** | Batch lookup-path hit counters if a profile names them | `head_resolve_stats::add_hit_rank` does two relaxed `fetch_add`s per resolved txid (`head_resolve_denserels`, `tx_table`). SH extract collect flushes output and hit counts once per fk batch, not per output. Still once per event: `accepted_wb` (`SeqCst`, per confirmed block), `serve_perf::note_serve` (per historical getdata), `page_ios` (per SH page read). Done: a profile shows `add_hit_rank` on confirm lookup and those counters flush per batch, or the profile shows the line is noise and this row moves to Won't-fix. `add_hit_ages` is already a batched flush; `add_hit_age` remains the one-bucket path. | R-ids were the 2026-08-12 slice. Canonical id is **bold**. Do not start -**R-11+**. Next unused Q-id is **Q-70**. +**R-11+**. Next unused Q-id is **Q-71**. Close work by **moving the Open row into CHANGELOG** in the same edit as the landing change (do not grow a Completed museum here). New item: insert