Skip to content

GH: Gandiva: link only the precompiled functions a query uses - #161

Open
lriggs wants to merge 1 commit into
dremio:dremio_27.0_23_19from
lriggs:gandiva-lazy-bitcode-link
Open

lriggs wants to merge 1 commit into
dremio:dremio_27.0_23_19from
lriggs:gandiva-lazy-bitcode-link

Conversation

@lriggs

@lriggs lriggs commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Gandiva links its entire precompiled bitcode library (~1600 functions) into every expression module and then relies on Internalize + global DCE to strip the unused ones. That link-and-prune runs on every uncached compile and is a large fixed cost independent of the expression.

Instead, parse the precompiled module lazily and add only declarations for its functions to the module up front; then at FinalizeModule use llvm::Linker::LinkOnlyNeeded to pull in just the function bodies the generated code actually references (and their dependencies), dropping declarations that went unused.

No change to generated code or results; this only reduces per-compile linking work. Cuts the fixed compile floor substantially on cold compiles.

@lriggs
lriggs force-pushed the gandiva-lazy-bitcode-link branch from 465ac88 to 17c3926 Compare November 14, 2026 01:06
Gandiva links its entire precompiled bitcode library (~1600 functions) into
every expression module and then relies on Internalize + global DCE to strip
the unused ones. That link-and-prune runs on every uncached compile and is a
large fixed cost independent of the expression.

Instead, parse the precompiled module lazily and add only *declarations* for
its functions to the module up front; then at FinalizeModule use
llvm::Linker::LinkOnlyNeeded to pull in just the function bodies the generated
code actually references (and their dependencies), dropping declarations that
went unused.

No change to generated code or results; this only reduces per-compile linking
work. Cuts the fixed compile floor substantially on cold compiles.
@lriggs
lriggs force-pushed the gandiva-lazy-bitcode-link branch from 17c3926 to ef89f61 Compare November 14, 2026 01:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant