Run the compilation in a container

[Edouard-chin]

Right now the compilation is done directly in the host. To compile gems on Linux this is problematic because it may end up compiling binaries that are incompatible with older linux versions.

For instance, if I compile the date gem on Ubuntu 24, then it's going to be incompatible on Ubuntu 22 and below. This is because the date gem makes use of strlcpy which is a new function added in glibc 2.38 (previously the function came from Ruby and exposed it conditionally using HAVE_STRLCPY).
If we run the compilation in a container we control the environment we are in and will be less prone to issues like this.

[eregon]

Right, I have run in a similar issue for TruffleRuby recently and the well-known solution seems to be to compile on an old Linux distro so that no newer symbols are picked from glibc.
I can confirm that works for my case (almost all failing when building on 22.04, all passing now).
It can be quite troublesome to compile on older Linux though due to various subtle differences, e.g. older gcc so might warn/error more or not support some features, might need explicit linking to pthread, RHEL 9's glibc can dynamically load libssl, etc 😓

From https://github.com/Shopify/cibuildgem?tab=readme-ov-file#existing-solutions

One of the most popular tool that enables to precompile binaries for different platform is the great rake-compiler-dock toolchain. It uses a cross compilation approach by periodically building docker images for various platforms and spinning up containers to compile the binaries.

As noted by @flavorjones, this toolchain works great but it's complex and brittle compared to the more simple process of compiling on the target platform.

Maybe it makes sense to just reuse rake-compiler-dock then?
"Just compiling on the target platform" doesn't work for GitHub Actions as the GitHub-hosted runner images are all pretty recent.

[Edouard-chin]

Thanks for having a look at this issue!

Maybe it makes sense to just reuse rake-compiler-dock then?

Hmmm, that's actually a very interesting idea 🤔. I think the biggest benefit is that the docker images from rake-compiler-dock are battle tested and this should make it trivial (last famous words) to plug it to cibuildgem. I'll give that a shot and see where this leads me. Thank you for the idea!

[kou]

Using rake-compiler-dock only for x86_64 glibc based Linux on x86_64 host may be an option because it's not a cross compilation. (We must avoid cross compilation. It requires very high maintenance cost.)

BTW, I feel that one binary for many Linux environments isn't realistic... There are many different factors such as libc kind, libc version, many base libraries (such as zlib and OpenSSL), ... In reality, I think that we need different binary for each different Linux environment.

[eregon]

Using rake-compiler-dock only for x86_64 glibc based Linux on x86_64 host may be an option because it's not a cross compilation. (We must avoid cross compilation. It requires very high maintenance cost.)

Right, so you're saying it's better to avoid cross-compilation (compiling for a different OS (in the broad sense, like Linux) or arch on some host machine), but fine to use rake-compiler-dock on Linux to compile against an older glibc.
On macOS and Windows you'd compile directly on the host system and use other tricks to make sure it works on older versions then?

rake-compiler-dock already supports cross compilation so I think that will keep working/be maintained.

BTW, I feel that one binary for many Linux environments isn't realistic... There are many different factors such as libc kind, libc version, many base libraries (such as zlib and OpenSSL), ... In reality, I think that we need different binary for each different Linux environment.

For glibc vs musl that difference is taken in account in RubyGems so those should be separate precompiled gems.

For libz the ABI has been stable forever so I don't think there is much of a concern (I tested TruffleRuby on various Linux distros, never needed to recompile the zlib extension, there is just one known ABI compat issue with the type of get_crc_table() that change, can easily be worked around it's a small static table).

For libssl/libcrypto yes it's a mess because many Linux distributions configure it differently, have different versions and ABI compat hasn't been great in the past at least (and notably doesn't work when doing things like have_func checks), basically any extension linking to OpenSSL cannot be precompiled (unless it statically links libssl+libcrypto, but that then makes it hard to update the "embedded" libssl/libcrypto).
I think we have to accept we cannot precompile gems depending on libssl directly (like puma), they can use the openssl gem though, that's fine.

Compiling a different binary for each different Linux environment sounds like hundreds of binaries because there are so many Linux distros and even if we "group" there are still so many variants in OpenSSL config & ABI version.

[Edouard-chin]

fine to use rake-compiler-dock on Linux [...] On macOS and Windows you'd compile directly on the host system

Yep, that's what I had in mind.

For glibc vs musl that difference is taken into account

👍 I also want to be able to let users ship binaries for linux musl, and rake-compiler-dock has a docker image available https://github.com/rake-compiler/rake-compiler-dock/pkgs/container/rake-compiler-dock-image

---

It would be great to be able to compile binarie for different linux, but this is going to be difficult until we modify the Rubygem specification like the one you guys reviewed in https://github.com/rubygems/rfcs/pull/60. Ultimately in the future if we could have Ruby alternative to manylinux, this would be ideal.

[kou]

On macOS and Windows you'd compile directly on the host system and use other tricks to make sure it works on older versions then?

On macOS: We can use -mmacosx-version-min but base libraries may be different in different macOS versions.

On Windows (RubyInstaller2): We can use WINVER and _WIN32_WINN for old Windows versions but base libraries may be different in different MSYS2 versions included in DevKit.

Compiling a different binary for each different Linux environment sounds like hundreds of binaries because there are so many Linux distros and even if we "group" there are still so many variants in OpenSSL config & ABI version.

So my opinion is "Goodby fat gem": https://rubykaigi.org/2025/presentations/ktou.html

Anyway, I can understand that fat gems' merit: Fast installation

In reality, I think that we can provide fat gems only for limited Linux environments.

It would be great to be able to compile binarie for different linux, but this is going to be difficult until we modify the Rubygem specification like the one you guys reviewed in rubygems/rfcs#60. Ultimately in the future if we could have Ruby alternative to manylinux, this would be ideal.

I'm positive for adding Ruby version information but I'm negative/neutral for adding platform information. I feel that we can't define platform information that satisfies all needs...

My idea (it's not tried yet...) is that we provide separated RubyGems server for each target such as "Ubuntu 22.04" and "RHEL 10".

Users specify a RubyGems server for fat gems in Gemfile:

source "https://rubygems.org"

source "https://rubygems.pkg.github.com/rubygems-ubuntu-22.04" do
  gem "extension-gem"
end

Fat gems maintainers (not gem developers) build fat gems for the specified platform (such as Ubuntu 22.04) automatically. RubyGems.org provides webhook ( https://guides.rubygems.org/rubygems-org-api/#webhook-methods ). So we can start building a gem when the gem is pushed automatically.