No sync shutdown path: nodedb_close during active sync intermittently crashes the process (SIGSEGV in sync connect path)

[emanzx]

Summary

There is no way to stop sync once nodedb_start_sync has been called, and nodedb_close during active sync intermittently crashes the process with SIGSEGV inside the sync connect path. The FFI discards the SyncClient that start_sync returns (nodedb-lite-ffi/src/sync.rs: let _sync_client = h.db.start_sync(config);), so the embedder has no handle to quiesce sync, and nodedb_close (open.rs: handle_registry::remove) drops the NodeDbHandle — db first, then the tokio Runtime — while the sync task may still be mid-poll.

Verified on origin/main @ ee9ccdd (cdylib built from that commit).

What happens

A minimal embedder sequence:

1. nodedb_open(":memory:", NULL)
2. nodedb_start_sync(h, "ws://127.0.0.1:1", "some-jwt") — deliberately unreachable origin, returns OK
3. nodedb_close(h)

crashes the process with SIGSEGV on roughly 6-10 of 10 runs when the host is a Crystal-runtime embedder (via the Crystal binding, which does nothing besides the three calls above). Backtrace of the core, all frames inside libnodedb_lite_ffi.so:

#0 nodedb_lite::nodedb::sync_delegate::delegate_impl::…::load_producer_state::{{closure}}
#1 nodedb_lite::sync::transport::connect::connect_and_run::{{closure}}
#2 nodedb_lite::nodedb::core::ops::crdt::…::start_sync::{{closure}}
#3 tokio::runtime::task::harness::Harness<T,S>::poll
#4 tokio::runtime::scheduler::multi_thread::worker::Context::run_task
…   (tokio multi_thread worker / blocking pool / std thread_start)

At crash time the main thread is parked inside nodedb_close → Runtime::drop → tokio::runtime::blocking::pool::BlockingPool::shutdown — i.e. the sync task is being polled on a worker while close is tearing the runtime down.

Two more observations from the core, offered as diagnosis input rather than conclusions:

• the faulting thread's rsp sits inside the guard page below its stack mapping, and the interrupted instruction is an ordinary store to its own frame; si_code is SI_KERNEL with si_addr = 0, which is the signature of failed signal-frame delivery on an exhausted stack rather than a plain bad pointer;
• the worker's outermost frames (thread_start → blocking::pool::Inner::run → worker::run) sit only ~20 KB above the stack floor, so the sync poll had almost no headroom before it started.

Host-dependence (reported for honesty)

A minimal pure-C harness doing the identical three calls against the same .so did not crash in 25 runs on the same machine, including variants that sleep 0.5 s before close, sleep 3 s after close, and run with MALLOC_PERTURB_ poisoning. The crash therefore needs something the richer host runtime provides (our leading hypothesis: signal traffic landing on the worker at peak stack depth, per the SI_KERNEL signature above). The crashing frames are nonetheless entirely library-internal, and the embedder's only API interaction was the documented three-call sequence.

Why it matters

Any embedder that starts sync and later closes the database — a test suite, a short-lived CLI, an app with a "log out" flow — is exposed to an intermittent process crash it cannot prevent, because there is no stop surface: the SyncClient is discarded inside the FFI, and close has no way to wait for the sync task. The only workaround is "once sync starts, never close the handle and let process exit clean up", which is what our binding now documents.

Suggested change

1. Retain the SyncClient in NodeDbHandle and make nodedb_close stop sync deterministically (signal + join/await the sync task) before dropping the runtime. That fixes the crash for every existing caller with no API change.
2. Optionally expose nodedb_stop_sync(handle) so embedders can quiesce sync without closing — useful for reconnect-with-new-token flows too.
3. Worth a look while in there: the stack headroom of the connect_and_run future on the worker it lands on (see the ~20 KB observation above) — even with teardown fixed, the connect path appears to run close to the limit.

Reproducing

Crystal host (crashes 6-10 of 10):

db = NodeDB::Lite.open(":memory:")      # nodedb_open(":memory:", NULL)
db.sync.start("ws://127.0.0.1:1", "x")  # nodedb_start_sync — OK (fire-and-forget)
db.close                                 # nodedb_close → intermittent SIGSEGV

Pure-C equivalent of the same three calls is attached in spirit above; on this machine it does not trigger the crash, so a Crystal (or comparably signal-active) host is currently the reliable reproducer. Core dumps consistently show the backtrace given above.