Skip to content

DX-128365: [C++][Gandiva] Add BuildScope::PreShutdown to wait for in-flight builds before exit - #163

Draft
victor6567758 wants to merge 1 commit into
dremio:dremio_27.0_23_19from
victor6567758:DX-128365-clean
Draft

victor6567758 wants to merge 1 commit into
dremio:dremio_27.0_23_19from
victor6567758:DX-128365-clean

Conversation

@victor6567758

Copy link
Copy Markdown

Summary

DX-128365: an executor crashed with SIGSEGV in LLVM code generation (buildProjector) during graceful shutdown. System.exit ends in exit(), which runs static destructors while builder threads are still compiling in Projector::Make / Filter::Make. A compile cannot be interrupted, so the embedder needs a way to stop new builds and wait for the running ones before the process exits.

This adds that, as an explicit call. There are no atexit hooks.

Changes

  • BuildScope (new, shutdown_guard.h/.cc): RAII scope held for the whole of Projector::Make (the overload that compiles) and Filter::Make. It counts builds in flight and rejects new ones once shutdown has started, with Status::Invalid("Gandiva is shutting down").
  • BuildScope::PreShutdown(timeout): rejects new builds and waits until none are in flight. Returns OK, IOError on timeout, or Invalid if called from inside a build (it would wait for itself).
  • The shutdown flag and the in-flight count share one atomic word, so "check flag and increment" is a single compare-and-swap. The mutex and condition variable are only used to put PreShutdown to sleep and wake it, so normal builds never take a lock.
  • Scopes nest per thread: only the outermost scope is counted, and nested scopes are always admitted.
  • The shared state is leaked on purpose so builds still running at exit find it alive.
  • shutdown_guard_test.cc, registered as its own test binary (gandiva-shutdown-guard-test) because PreShutdown is one-way for the life of the process.

Why compiles are covered

Gandiva starts no threads. LLJITBuilder is used with the default NumCompileThreads = 0, and the machine code is generated on the first symbol lookup, which LLVMGenerator::Build performs for every expression inside Make. So the whole compile happens on the calling thread, inside the scope.

Follow-up

A JNI binding and a call from Dremio's executor shutdown (arrow-java and Dremio changes) will come in separate PRs.

Testing

  • gandiva-shutdown-guard-test
  • existing Gandiva tests (ctest -R gandiva)
  • Repro: a C++ driver that keeps building projectors on 4 threads and calls exit() at a random time. Without PreShutdown it crashed in about 8 of 20 runs. With PreShutdown(30s) before exit(), the runs completed with PreShutdown: OK and exit code 0. Final counts to be added before undrafting.

@lriggs

lriggs commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Should this handle other cases besides build? like evaluate or shutdown?

@lriggs

lriggs commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Also the status code is ambiguous. Status::Invalid is also what a bad expression returns. Status::Cancelled (or a unique message the JNI layer can map) would let callers tell them apart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants