`trie-db` 0.31.0 causes huge benchmark regression for storage-heavy extrinsics

[sigurpol]

The problem

We have observed a huge regression in benchmark execution time once bumping from trie-db 0.30.0 to 0.31.0 from benchmark whose setup does massive storage operations.

This is critical because we can't confidently benchmark some pallets for the upcoming 2.1.0 Polkadot / Kusama release.

A concrete example

Let's take a simple dummy extrinsic as example:

force_apply_min_commission (here) is a simple extrinsic doing 2read and 1 write so having ms as execution time doesn't make sense.

What is peculiar in the specific benchmark (and in many in staking-async pallet in the SDK) is that before benchmarking, we do massive storage deletion (e.g. on PolkadotAH we delete 27k nominators/ validators and all related storage items - so a lot). This happens in the benchmark setup phase and shouldn't therefore leak into benchmark results (see Notes at the end for a recent fix/workaround in frame-omni-bencher).

force_apply_min_commission in pallet_staking_async takes ~46µs with trie-db 0.30.0 but ~2.8ms with trie-db 0.31.0 while testing on my Ubuntu desktop. Results are aligned with what we see while running benchmark on fellowship CI (QEMU/native runner): look at the difference in results between a frame-omni-bencher built with 0.30.0 and one with 0.31.0 here -> polkadot-fellows/runtimes#1065 (comment)

The regression affects any benchmark whose setup (so before actually measuring the extrinsic) populates/deletes a large trie (e.g. > 25k validator/nominator entries) — the overhead seems to come from commit_db() / trie backend operations, not in the benchmarked extrinsic itself.

Bisection summary

(results coming from my local desktop, we observed the same on CI)

• frame-omni-bencher v0.15.0 / v0.16.0 (trie-db 0.30.0) → ~46 µs
• frame-omni-bencher v0.17.0+ (trie-db 0.31.0, PR bump trie-db version to 0.31.0 polkadot-sdk#10573) → ~3 ms
• Same runtime WASM, different frame-omni-bencher binary — confirms it's an issue with the binary

Root cause: trie-db bump 0.30.0 → 0.31.0 (#226 #226) merged in polkadot-sdk#10573 (2025-12-11) paritytech/polkadot-sdk#10573

The relationship between frame-omni-bencher and trie-db

We are using frame-omni-bencher to benchmark pallet extrinsics. For staking-async pallet many benchmark setup implies bulk storage operation like the deletion of thousands of items from storage (e.g. 26k nominators/validators created at genesis).
Debatable or not, this is not the point now.

trie-db is not a direct dependency of frame-omni-bencher
AFAIK the chain is the following:

  frame-omni-bencher                                                            
    → frame-benchmarking-cli                                                
      → sc-executor (runs WASM benchmarks)                                      
        → sp-state-machine (storage overlay + trie backend)                 
          → sp-trie                                                             
            → trie-db 

During benchmarking, the host creates an in-memory trie backend (via sp-state-machine) to hold the genesis state.
Every storage read/write during benchmark setup (like clear_validators_and_nominators() inserting/removing dozen thousands of entries) and commit_db() goes through sp-trie → trie-db on the host side.

The regression is in this host-side trie layer — the trie operations that materialize the storage overlay into the backend before the benchmark timing starts.
The benchmarked extrinsic itself is fast (1 read - 2 write in my example) ; it's the trie commit of the massive genesis state that got much slower with trie-db 0.31.0.
This shouldn't leak into benchmark results in any way - see paritytech/polkadot-sdk#10798 and in particular this comment from @cheme about writing I/O operation still on going after commit returns early.

Reproduction steps

Checkout latest main from https://github.com/polkadot-fellows/runtimes.
Then from there:

# build polkadot asset hub
cargo build -p asset-hub-polkadot-runtime --profile production --features runtime-benchmarks
# install latest and greatest frame-omni-bencher
cargo install frame-omni-bencher
# test with frame-omni-bencher  the specific extrinsic 
frame-omni-bencher v1 benchmark pallet --runtime ./target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm --pallet pallet_staking_async --extrinsic "force_apply_min_commission" --steps 2 --repeat 1
## You will see execution time for this extrinsics around few ms

# Now install frame-omni-bencher v0.16.0 => the latest with trie-db 0.30.0
cargo install frame-omni-bencher --version 0.16.0 // if your clang/gcc are very recent, prefix with CXXFLAGS="-include cstdint"
# test with frame-omni-bencher 
frame-omni-bencher v1 benchmark pallet --runtime ./target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm --pallet pallet_staking_async --extrinsic "force_apply_min_commission" --steps 2 --repeat 1
## You will see execution time for this extrinsics around few microseconds!

Notes

The issue happens on Polkadot and Kusama AssetHub in the fellowship runtime repo. Interestingly enough, when running the same benchmark on Westend AssetHub within SDK using the same frame-omni-bencher, the results is in the µs ball-park and not ms.
To be noted that on Westend AssetHub (see paritytech/polkadot-sdk#10798), we have observed initially execution time for this benchmark (massive storage deletion in setup + 1 read - 2 write in execution) around fw ms but that was "fixed" doing a dummy DB read/write in SDK PR paritytech/polkadot-sdk#10802 and then (better) paritytech/polkadot-sdk#10974 - this fix has brought down execution time to µs on Westend AssetHub (Where we also do bulk deletion before executing the extrinsic in the benchmark) - but not on Polkadot / Kusama AssetHub.

[sigurpol]

cc @cheme , @bkchr , @kianenigma - followup of what we discussed in the scope of SDK #10798 - the dummy read/write in the setup was good enough for Westend AssetHub but it's not for Polkadot and Kusama AssetHub (well, without the dummy r/w results degrade of other few ms so it's not completely pointless 😅 )

[sigurpol]

Might very well be that the issue is not in the trie-db itself but we might (ab)use of it indirectly in frame-omni-bencher / chain of dependencies below (sp-state-machine / sp-trie ...) - I am just creating the issue here to start the investigation since if we build frame-omni-bencher with trie-db 0.30 the problem doesn't happen.

[cheme]

I had a quick look at all PR on trie crate between 0.30 and 0.31, and I really see nothing that could result in big regression.
Only potential impact is from switching from btreemap to hashmap but it is only for no_std so not host side. Reverting #221 can confirm this.
Are we sure that running with trie-db 0.30 is making things faster (in this case I would think regression is more on guest/wasm side (I don’t remember if a memory db could be use in guest side somewhere)).

Edit: @sigurpol I did not receive your previous comment when typing this, when you say with 0.30, did you revert the whole paritytech/polkadot-sdk#10573 or just patch trie version and manually resolve api change?
Edit 2: @sigurpol please forget about my first edit, I was lazy to look at 10573, but yes there is no api incompatibility so same as just switching

[sigurpol]

@cheme I have built manually frame-omni-bencher with the commit before #10573 and exactly with the commit associated to #10573 via cargo build -p frame-omni-bencher --release so to have only #10573 as difference and the result is that with the former execution time is microseconds for my dummy extrinsics, and for the latter milliseconds.

so for SDK:

git checkout 23cac32cb789a6a6e3a071a3f5cd293737098b71 ->
HEAD is now at 23cac32cb7 [CI] `All * passed` jobs check if cancelled (#10566)
# build frame-omni-builder and then run benchmark for runtime like in reproduction step
git checkout 7b40bd275e2497bf06c4fe4e5edb6fdac0e3deb4
Previous HEAD position was 23cac32cb7 [CI] `All * passed` jobs check if cancelled (#10566)
HEAD is now at 7b40bd275e bump trie-db version to 0.31.0 (#10573)
# rebuild frame-omni-bencher and re-run benchmark

Edit: I have also added the reproduction steps now in the ticket, the same is observable with "official" frame-omni-bencher with 0.16.0 latest published version with 0.30.0 for trie-db, and 0.17.0 first version with 0.31.0

Edit2:

In both cases I am running the test against the same PolkadotAH WASM

frame-omni-bencher v1 benchmark pallet --runtime ./target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm --pallet pallet_staking_async --extrinsic "force_apply_min_commission" --steps 2 --repeat 1

Edit3:

Results with the two frame-omni-bencher variant below:

### FRAME-OMNI-BENCHER built from SDK commit 23cac32cb7 (HEAD) [CI] `All * passed` jobs check if cancelled (#10566) - the commit before #10573
/home/paolo/github/polkadot-sdk/target/release/frame-omni-bencher v1 benchmark pallet --runtime ./target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm --pallet pallet_staking_async --extrinsic "force_apply_min_commission" --steps 2 --repeat 1
2026-02-13T10:17:39.176194Z  INFO frame::benchmark::pallet: Initialized runtime log filter to 'INFO'
2026-02-13T10:17:40.791691Z  INFO pallet_collator_selection::pallet: assembling new collators for new session 0 at #0
2026-02-13T10:17:40.791720Z  INFO pallet_collator_selection::pallet: assembling new collators for new session 1 at #0
2026-02-13T10:19:40.953618Z  INFO frame::benchmark::pallet: [  0 % ] Starting benchmark: pallet_staking_async::force_apply_min_commission
Pallet: "pallet_staking_async", Extrinsic: "force_apply_min_commission", Lowest values: [], Highest values: [], Steps: 2, Repeat: 1
Raw Storage Info
========
Storage: `Staking::MinCommission` (r:1 w:0)
Proof: `Staking::MinCommission` (`max_values`: Some(1), `max_size`: Some(4), added: 499, mode: `MaxEncodedLen`)
Storage: `Staking::Validators` (r:1 w:1)
Proof: `Staking::Validators` (`max_values`: None, `max_size`: Some(45), added: 2520, mode: `MaxEncodedLen`)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    50.57
              µs

Reads = 2
Writes = 1
Recorded proof Size = 564

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    50.57
              µs

Reads = 2
Writes = 1
Recorded proof Size = 564

### FRAME-OMNI-BENCHER built from SDK commit 7b40bd275e (HEAD) bump trie-db version to 0.31.0 (#10573)
➜  runtimes git:(sigurpol-release-2_0_6) /home/paolo/github/polkadot-sdk/target/release/frame-omni-bencher v1 benchmark pallet --runtime ./target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm --pallet pallet_staking_async --extrinsic "force_apply_min_commission" --steps 2 --repeat 1
2026-02-13T10:26:40.222012Z  INFO frame::benchmark::pallet: Initialized runtime log filter to 'INFO'
2026-02-13T10:26:41.784199Z  INFO pallet_collator_selection::pallet: assembling new collators for new session 0 at #0
2026-02-13T10:26:41.784228Z  INFO pallet_collator_selection::pallet: assembling new collators for new session 1 at #0
2026-02-13T10:29:23.435076Z  INFO frame::benchmark::pallet: [  0 % ] Starting benchmark: pallet_staking_async::force_apply_min_commission
Pallet: "pallet_staking_async", Extrinsic: "force_apply_min_commission", Lowest values: [], Highest values: [], Steps: 2, Repeat: 1
Raw Storage Info
========
Storage: `Staking::MinCommission` (r:1 w:0)
Proof: `Staking::MinCommission` (`max_values`: Some(1), `max_size`: Some(4), added: 499, mode: `MaxEncodedLen`)
Storage: `Staking::Validators` (r:1 w:1)
Proof: `Staking::Validators` (`max_values`: None, `max_size`: Some(45), added: 2520, mode: `MaxEncodedLen`)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=     5034
              µs

Reads = 2
Writes = 1
Recorded proof Size = 564

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=     5034
              µs

Reads = 2
Writes = 1
Recorded proof Size = 564

[cheme]

can exclude all btreemap/hashmap consideration as it touches memory-db that #10573 do not update (wrong direction).

[cheme]

@sigurpol just to be sure, do you have a diff of both cargo .lock in your previous tests (pre 10573 and post 10573), I am still wondering if maybe memory-db could have been updated by being pulled from trie-bench?

[sigurpol]

memory-db has not been updated, see below:

git diff  23cac32cb7..7b40bd275e2497bf06c4fe4e5edb6fdac0e3deb4 -- Cargo.lock
diff --git a/Cargo.lock b/Cargo.lock
index d129f4eea4..385761b7b8 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1409,6 +1409,7 @@ dependencies = [
  "pallet-indices",
  "pallet-message-queue",
  "pallet-migrations",
+ "pallet-multi-asset-bounties",
  "pallet-multisig",
  "pallet-nft-fractionalization",
  "pallet-nfts",
@@ -5128,6 +5129,8 @@ dependencies = [
  "cumulus-primitives-core",
  "parity-scale-codec",
  "polkadot-primitives",
+ "sp-consensus-babe",
+ "sp-core 28.0.0",
  "sp-runtime",
  "sp-state-machine",
  "sp-trie",
@@ -14419,6 +14422,7 @@ dependencies = [
  "pallet-balances",
  "pallet-collator-selection",
  "pallet-message-queue",
+ "pallet-multi-asset-bounties",
  "pallet-treasury",
  "pallet-xcm",
  "parity-scale-codec",
@@ -23331,6 +23335,7 @@ dependencies = [
 name = "sp-debug-derive"
 version = "14.0.0"
 dependencies = [
+ "proc-macro-warning",
  "proc-macro2 1.0.95",
  "quote 1.0.40",
  "syn 2.0.98",
@@ -26339,9 +26344,9 @@ dependencies = [

 [[package]]
 name = "trie-bench"
-version = "0.42.0"
+version = "0.42.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "972be214c558b1a5550d34c8c7e55a284f6439cefc51226d6ffbfc152de5cc58"
+checksum = "a03bee4700c5dd6b2ceba5e4e4d5a7017704a761481824d3033d223f9660973a"
 dependencies = [
  "criterion",
  "hash-db",
@@ -26355,9 +26360,9 @@ dependencies = [

 [[package]]
 name = "trie-db"
-version = "0.30.0"
+version = "0.31.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6c0670ab45a6b7002c7df369fee950a27cf29ae0474343fd3a15aa15f691e7a6"
+checksum = "a7795f2df2ef744e4ffb2125f09325e60a21d305cc3ecece0adeef03f7a9e560"
 dependencies = [
  "hash-db",
  "log",

[alexggh]

Had a quick look the difference between: git log --oneline 23cac32cb7..7b40bd275e, is this:

7b40bd275e2 bump trie-db version to 0.31.0 (#10573)
f8c964316e0 Deprecate `RuntimeDebug` and replace it with `Debug` (#10582)
864da3057d0 Enable `trace` logging for sync in the `full_node_warp_sync` test (#10586)
6e3d8ba7a34 Add `pallet_multi_asset_bounties` to Westend Asset Hub testnet runtime. (#10064)
0f0f0aeb954 [Release|CI/CD] Fix Create polkadot-vX.YY.Z tag flow (#10551)
18cb5ee6196 Align Types between Bulletin and SDK (#10578)
acd03062aeb fatxpool: metrics for unfiltered timings of inblock events (#10312)
7ef17510dd3 sc-client-db: Make `jemalloc` optional (#10590)
0f28d8104be test-utils/fix: Parachains test-utils relay parent descendants mock data (#10541)
2a1f0b257c4 [Revive] Fix construction of negative zero SignedGas (#10567)
30fe555f43a Increase the omni-bencher CI checks timeout (#10594)
19307dd5f8a Fix runtime migrations check (#10570)

So I got puzzled by this, I can pinpoint the problem appearing after

7ef17510dd3 sc-client-db: Make `jemalloc` optional (#10590)

Performance on master and runtime built from polkadot-fellows/runtimes#1065:

Goes from

-- Extrinsic Time --

Model:
Time ~=     1992
              µs

To

-- Extrinsic Time --

Model:
Time ~=    29.33
              µs

I fixed it with something like, feel free to open a PR in polkadot-sdk.

diff --git a/Cargo.lock b/Cargo.lock
index 12d8574d375..8feadaa07f5 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -6845,6 +6845,7 @@ dependencies = [
  "frame-benchmarking-cli",
  "sc-chain-spec",
  "sc-cli",
+ "sc-client-db",
  "sp-genesis-builder",
  "sp-runtime",
  "sp-statement-store",
diff --git a/substrate/utils/frame/omni-bencher/Cargo.toml b/substrate/utils/frame/omni-bencher/Cargo.toml
index 5d9cf766b32..a685361c39b 100644
--- a/substrate/utils/frame/omni-bencher/Cargo.toml
+++ b/substrate/utils/frame/omni-bencher/Cargo.toml
@@ -16,6 +16,7 @@ clap = { features = ["derive"], workspace = true }
 cumulus-primitives-proof-size-hostfunction = { workspace = true, default-features = true }
 frame-benchmarking-cli = { workspace = true }
 sc-cli = { workspace = true, default-features = true }
+sc-client-db = { features = ["jemalloc-allocator"], workspace = true }
 sp-runtime = { workspace = true, default-features = true }
 sp-statement-store = { workspace = true, default-features = true }
 tracing-subscriber = { workspace = true }

FYI @sigurpol , @cheme

[cheme]

I am wondering (in the cargo lock diff) , why we onboard new crates such as "pallet-multi-asset-bounties" or "sp-core 0.28.0".

Missing jemalloc could indeed explain this (but I would expect cargo lock to show a change in sc-client-db).

[alexggh]

I am wondering (in the cargo lock diff) , why we onboard new crates such as "pallet-multi-asset-bounties" or "sp-core 0.28.0".

Missing jemalloc could indeed explain this (but I would expect cargo lock to show a change in sc-client-db).

No idea, but I'm pretty sure sc-client-db: Make `jemalloc` optional (#10590), is the one that introduced the slowness.

Maybe @bkchr knows why nothing shows in the Cargo.lock.