PhysicsPipeline::step hangs in join_deferred_bvh_optimize with parallel enabled

[lelierre-dev]

I'm seeing a physics freeze with Rapier's parallel feature. I first hit it in a Godot game on an iPhone, then reproduced it on macOS with the standalone Rust program below. The Rust program does not use Godot or any game code.

The reproducer uses rapier3d 0.35.1 with parallel and enhanced-determinism, rayon 1.12.0 and rayon-core 1.13.0. I tested it on an M1 Pro running macOS 26.1, with rustc 1.98.0.

The original game freezes were on an iPhone 17 Pro Max running iOS 26.6.1, Godot 4.7.1 and godot-rapier-physics v0.35.2. That addon bundles rapier3d 0.35.1 and uses a pool of 2 workers on this phone. It calls the step through pool.install(|| pipeline.step(...)), which is also how the reproducer runs it.

When it hangs, the main thread is waiting for ThreadPool::install to return. One worker is blocked in join_deferred_bvh_optimize, waiting on the channel. The other worker is asleep in Rayon's scheduler. This is the relevant part of the stacks, simplified for readability:

Main thread:
  ThreadPool::install
  Registry::in_worker_cold
  LockLatch::wait_and_reset
  _pthread_cond_wait

Worker running the step:
  PhysicsPipeline::step_inner
  PhysicsPipeline::join_deferred_bvh_optimize
  Receiver<DeferredBvhOptimize>::recv
  Thread::park
  semaphore_wait_trap

Other worker:
  WorkerThread::wait_until_cold
  Sleep::sleep
  _pthread_cond_wait

Both iPhone reports show this pattern. A fresh sample of the frozen Rust program shows it too. The attachment contains selected frames from the reports, with local paths and process identifiers removed.

To reproduce, create the two files below and run:

cargo run --release -- 2 40

The program adds 100 moving cuboids every 300 steps and removes them 200 steps later. It also has a ground collider and 30 fixed boxes. A watchdog reports a hang if no step completes for 8 seconds, then exits with code 2. The second argument is the normal run duration; the watchdog can run past that duration if a step is stuck. HOLD=1 keeps the process alive after detection so it can be sampled.

Cargo.toml

[package]
name = "repro_bvh_deadlock"
version = "0.1.0"
edition = "2021"

[dependencies]
rapier3d = { version = "=0.35.1", features = ["parallel", "enhanced-determinism"] }
rayon = "1"

[profile.release]
opt-level = 3

src/main.rs

use rapier3d::prelude::*;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};

fn main() {
    let args: Vec<String> = std::env::args().collect();
    let threads: usize = args.get(1).and_then(|s| s.parse().ok()).unwrap_or(2);
    let max_secs: u64 = args.get(2).and_then(|s| s.parse().ok()).unwrap_or(240);
    let burst: usize = args.get(3).and_then(|s| s.parse().ok()).unwrap_or(100);
    let period: u64 = args.get(4).and_then(|s| s.parse().ok()).unwrap_or(300);
    let pool = rayon::ThreadPoolBuilder::new().num_threads(threads).build().unwrap();
    println!("pool threads={} max_secs={} burst={} period={}", threads, max_secs, burst, period);

    let progress = Arc::new(AtomicU64::new(0));
    {
        let progress = progress.clone();
        std::thread::spawn(move || {
            let mut last = 0u64;
            let mut stuck_since = Instant::now();
            loop {
                std::thread::sleep(Duration::from_millis(200));
                let now = progress.load(Ordering::SeqCst);
                if now != last { last = now; stuck_since = Instant::now(); }
                else if stuck_since.elapsed() > Duration::from_secs(8) {
                    eprintln!("DEADLOCK: no step completed for 8 s, stuck at step {}", last);
                    if std::env::var("HOLD").is_ok() { loop { std::thread::sleep(Duration::from_secs(1)); } }
                    std::process::exit(2);
                }
            }
        });
    }

    let mut pipeline = PhysicsPipeline::new();
    let gravity = Vector::new(0.0, -9.81, 0.0);
    let params = IntegrationParameters::default();
    let mut islands = IslandManager::new();
    let mut bp = DefaultBroadPhase::new();
    let mut np = NarrowPhase::new();
    let mut bodies = RigidBodySet::new();
    let mut colliders = ColliderSet::new();
    let mut ij = ImpulseJointSet::new();
    let mut mj = MultibodyJointSet::new();
    let mut ccd = CCDSolver::new();
    let ground = bodies.insert(RigidBodyBuilder::fixed());
    colliders.insert_with_parent(ColliderBuilder::cuboid(200.0, 0.5, 200.0), ground, &mut bodies);
    // Fixed boxes to give the broad phase some static geometry.
    for i in 0..30 {
        let h = bodies.insert(RigidBodyBuilder::fixed().translation(Vector::new((i as f32) * 3.0 - 45.0, 2.0, 10.0)));
        colliders.insert_with_parent(ColliderBuilder::cuboid(1.0, 2.0, 1.0), h, &mut bodies);
    }
    let mut seed: u64 = 0x5E17;
    let mut rnd = move || { seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407); ((seed >> 33) as f32) / (u32::MAX as f32 / 2.0) };
    let mut debris: Vec<RigidBodyHandle> = Vec::new();
    let start = Instant::now();
    let mut step: u64 = 0;
    loop {
        if step % period == 0 {
            for _ in 0..burst {
                let h = bodies.insert(RigidBodyBuilder::dynamic()
                    .translation(Vector::new(rnd() * 20.0 - 10.0, 6.0 + rnd() * 4.0, rnd() * 20.0 - 10.0))
                    .linvel(Vector::new(rnd() * 20.0 - 10.0, rnd() * 10.0, rnd() * 20.0 - 10.0))
                    .angvel(Vector::new(rnd() * 6.0, rnd() * 6.0, rnd() * 6.0)));
                colliders.insert_with_parent(ColliderBuilder::cuboid(0.2 + rnd() * 0.3, 0.2 + rnd() * 0.3, 0.2 + rnd() * 0.3), h, &mut bodies);
                debris.push(h);
            }
        }
        if step % period == period * 2 / 3 {
            for h in debris.drain(..) {
                bodies.remove(h, &mut islands, &mut colliders, &mut ij, &mut mj, true);
            }
        }
        pool.install(|| {
            pipeline.step(gravity, &params, &mut islands, &mut bp, &mut np, &mut bodies, &mut colliders, &mut ij, &mut mj, &mut ccd, &(), &());
        });
        step += 1;
        progress.store(step, Ordering::SeqCst);
        if step % 20000 == 0 {
            println!("{} steps, {:.0} s, {} bodies", step, start.elapsed().as_secs_f32(), bodies.len());
        }
        if start.elapsed() > Duration::from_secs(max_secs) {
            println!("NO DEADLOCK after {} steps in {} s", step, max_secs);
            break;
        }
    }
}

These are the results collected on this Mac, including three fresh runs with 2 workers and a longer check with 1 worker:

Workers	Runs	Result
1	2	No hang in runs of 20 and 60 seconds. The latter completed 945,614 steps.
2	11	All hung, after 913 to 53,190 completed steps.
3	2	Both hung, after 24,658 and 27,386 steps.
4	2	Both hung, after 288,713 and 364,916 steps.
6	2	One completed 30 seconds; the longer run hung after 337,862 steps.
8	1	Hung after 975,736 steps.

These runs show that the hang can occur with more than 2 workers. There aren't enough runs to establish a failure rate for each pool size. With 1 worker, Rapier runs the deferred optimization inline at the join point instead of using the channel. I haven't seen a hang in that configuration during these tests.

Looking at the scheduling code in 0.35.1, the deferred optimization is submitted with rayon::spawn. The step later waits for it with a blocking recv() in join_deferred_bvh_optimize. That wait prevents the current worker from running other queued work. I suspect this combination, but the stacks alone don't explain why the other workers remain asleep or prove a lost wakeup. I haven't tested a fix.

The same scheduling and receive code is present in the 0.35.3 sources I checked. The Godot addon's v0.35.4 lockfile still lists rapier3d 0.35.1. I have only run this reproducer against rapier3d 0.35.1.

piles-a-joindre.txt
rapier-bvh-hang-reproducer.zip

[NectarZuzi]

Confirming this on Windows 10 x86_64

Environment

godot-rapier2d 0.35.4 (appsinacup, single-precision build, parallel + enhanced-determinism), Godot 4.7.2 stable, Windows 10 x86_64.
Symptom: intermittent hard freeze during PhysicsServer2D stepping (an ordinary 2D step, seen as early as the first physics-heavy frame). The process sits at 0% CPU until externally killed. In an automated sweep I caught 5 frozen processes concurrently; frequency is on the order of 1 in ~100 runs.
Thread topology matches yours exactly. The shipped DLL has no public symbols, so my frames are module+offset. The addon spawns four rapier-worker threads, and three separately-captured minidumps were byte-for-byte identical:

main thread — parked on the pool join (your ThreadPool::install → LockLatch::wait_and_reset).
rapier-worker-0 — deep inside the step, blocked on the deferred-BVH channel (your join_deferred_bvh_optimize → Receiver::recv → Thread::park).
rapier-worker-1/2/3 — idle, parked at the pool wait (your WorkerThread::wait_until_cold → Sleep::sleep).
Exactly the lost-wakeup you describe.