Skip to content

fix(ci): repair cross image builds #6617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
fmassot wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

fix(ci): repair cross image builds #6617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ce3a204
fix(ci): repair cross image builds
fmassot Jul 19, 2026
45dcddb
fix(ci): expose Lambda artifact to cross builds
fmassot Jul 19, 2026
a8e48a9
fix(ci): add curl headers to GNU cross images
fmassot Jul 19, 2026
aac5f29
fix(ci): expose ARM64 Zlib to CMake
fmassot Jul 19, 2026
6cd3951
fix(ci): preserve Cargo CMake dependency prefixes
fmassot Jul 19, 2026
be0ffc3
fix(ci): link resolver for ARM64 releases
fmassot Jul 19, 2026
d2f17c2
fix(ci): preserve ARM64 release Rust flags
fmassot Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .cargo/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,13 @@ rustflags = ["-C", "target-cpu=x86-64-v3", "--cfg", "tokio_unstable"]
# Targeting neoverse-n1 covers AWS Graviton2+ and GCP T2A (Ampere Altra).
# Key gains over generic ARMv8-A: LSE atomics (better concurrency), dot
# product instructions, and ARMv8.2 crypto extensions.
rustflags = ["-C", "target-cpu=neoverse-n1", "--cfg", "tokio_unstable"]

# sasl2-sys omits libresolv when HOST != TARGET, but its vendored Kerberos
# archive still references resolver symbols when cross-compiling for Linux.
rustflags = [
"-C",
"target-cpu=neoverse-n1",
"-C",
"link-arg=-lresolv",
"--cfg",
"tokio_unstable",
]
8 changes: 4 additions & 4 deletions .github/workflows/publish_release_packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,12 +107,12 @@ jobs:
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: ${{ needs.build-lambda.outputs.artifact-name }}
# Must be within the workspace so cross can mount it into the build container.
path: ./lambda-artifact
# `cross` mounts the Rust workspace (`quickwit/`) at `/project` in the container.
path: ./quickwit/lambda-artifact
- name: Set LAMBDA_ZIP_PATH
run: |
ZIP=$(ls lambda-artifact/*.zip)
echo "LAMBDA_ZIP_PATH=$(pwd)/$ZIP" >> $GITHUB_ENV
ZIP=$(ls quickwit/lambda-artifact/*.zip)
echo "LAMBDA_ZIP_PATH=/project/lambda-artifact/$(basename "$ZIP")" >> "$GITHUB_ENV"
- uses: ./.github/actions/cross-build-binary
with:
target: ${{ matrix.target }}
Expand Down
3 changes: 2 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ IMAGE_TAGS = x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu x86_64-unknown-l

.PHONY: cross-images
cross-images:
@for tag in ${IMAGE_TAGS}; do \
@set -e; \
for tag in ${IMAGE_TAGS}; do \
docker build --tag quickwit/cross:$$tag --file ./build/cross-images/$$tag.dockerfile ./build/cross-images; \
docker push quickwit/cross:$$tag; \
done
Expand Down
16 changes: 16 additions & 0 deletions build/cross-images/aarch64-unknown-linux-gnu.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,30 @@ ARG PBC_URL="https://github.com/protocolbuffers/protobuf/releases/download/v21.5
# `rdkafka/gssapi-vendored` feature when there is a release including:
# https://github.com/MaterializeInc/rust-sasl/pull/48

# librdkafka 2.12.1 includes curl/curl.h even when OIDC support is disabled.
RUN dpkg --add-architecture arm64 && \
apt-get update && \
apt-get install -y --no-install-recommends \
binutils-aarch64-linux-gnu \
g++-10-aarch64-linux-gnu \
gcc-10-aarch64-linux-gnu \
libcurl4-openssl-dev:arm64 \
libsasl2-dev:arm64 \
unzip && \
rm -rf /var/lib/apt/lists/*

# cmake-rs passes vendored dependency prefixes in the CMAKE_PREFIX_PATH
# environment variable. Materialize that colon-separated value before the
# cross-rs toolchain constructs its target-only search roots.
RUN sed -i '/set(CMAKE_FIND_ROOT_PATH /i\ string(REPLACE ":" ";" CMAKE_PREFIX_PATH "$ENV{CMAKE_PREFIX_PATH}")' /opt/toolchain.cmake

# GCC 9.4 is affected by https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95189.
# aws-lc cannot execute its compiler probe while cross-compiling, so select the
# verified GCC 10.5 toolchain explicitly for both compilation and linking.
ENV CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc-10 \
CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++-10 \
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc-10
Comment on lines +35 to +37

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Point CMake at the GCC 10 cross compiler

For the ARM64 GNU release build, these variables only switch Cargo/cc-rs and the final linker; CMake-based crates still inherit cross-rs's toolchain file, which sets CMAKE_C_COMPILER from CROSS_TOOLCHAIN_PREFIX as ${prefix}gcc, and this image leaves that at the base default. Since .github/actions/cross-build-binary/action.yml builds release-feature-vendored-set and rdkafka-sys is enabled with cmake-build, librdkafka is still compiled with the old /usr/bin/aarch64-linux-gnu-gcc instead of aarch64-linux-gnu-gcc-10, so the GCC 9.4 compiler bug this image is trying to avoid remains for the ARM64 Linux artifact unless the CMake toolchain prefix/suffix or compiler is updated too.

Useful? React with 👍 / 👎.


RUN curl -fLO $PBC_URL && \
unzip protoc-21.5-linux-x86_64.zip -d ./protobuf && \
mv ./protobuf/bin/protoc /usr/bin/ && \
Expand Down
5 changes: 4 additions & 1 deletion build/cross-images/aarch64-unknown-linux-musl.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ FROM rustembedded/cross:aarch64-unknown-linux-musl@sha256:22627e0ba533781062127b
# ALSO UPDATE hooks/build!
ARG OPENSSL_VERSION=1.1.1i
ARG ZLIB_VERSION=1.2.11
ARG ZLIB_SHA256=629380c90a77b964d896ed37163f5c3a34f6e6d897311f1df2a7016355c45eff

RUN echo "Building OpenSSL" && \
cd /tmp && \
Expand All @@ -26,7 +27,9 @@ RUN echo "Building OpenSSL" && \

RUN echo "Building zlib" && \
cd /tmp && \
curl -fLO "https://zlib.net/fossils/zlib-$ZLIB_VERSION.tar.gz" && \
curl -fL --retry 3 -o "zlib-$ZLIB_VERSION.tar.gz" \
"https://github.com/madler/zlib/archive/refs/tags/v$ZLIB_VERSION.tar.gz" && \
echo "$ZLIB_SHA256 zlib-$ZLIB_VERSION.tar.gz" | sha256sum --check - && \
tar xzf "zlib-$ZLIB_VERSION.tar.gz" && cd "zlib-$ZLIB_VERSION" && \
AR=aarch64-linux-musl-ar CC=aarch64-linux-musl-gcc ./configure --static --prefix=/usr/local/aarch64-linux-musl && \
make && make install && \
Expand Down
10 changes: 10 additions & 0 deletions build/cross-images/x86_64-unknown-linux-gnu.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,22 @@ FROM ghcr.io/cross-rs/x86_64-unknown-linux-gnu:main@sha256:2431cbfcf2499f8a00570

ARG PBC_URL="https://github.com/protocolbuffers/protobuf/releases/download/v21.5/protoc-21.5-linux-x86_64.zip"

# librdkafka 2.12.1 includes curl/curl.h even when OIDC support is disabled.
RUN apt-get update && \
apt-get install -y --no-install-recommends \
g++-10 \
gcc-10 \
libcurl4-openssl-dev \
libsasl2-dev \
unzip && \
rm -rf /var/lib/apt/lists/*
Comment on lines 17 to 18

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve CMake dependency prefixes for x86 builds

When the x86_64 Linux release builds release-feature-vendored-set, libz-sys/static builds zlib and rdkafka-sys registers it as a CMake dependency, but cmake-rs passes that prefix via the CMAKE_PREFIX_PATH environment variable while the cross-rs toolchain consumes the CMake variable in CMAKE_FIND_ROOT_PATH. This file never materializes the env value for the x86 image (and no longer installs zlib1g-dev), so find_package(ZLIB) in librdkafka's CMake build is limited to the cross sysroot and misses the vendored zlib prefix, breaking the x86_64 GNU artifact unless the same prefix preservation is applied here.

Useful? React with 👍 / 👎.


# GCC 9.4 is affected by https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95189,
# which aws-lc correctly rejects because it can miscompile memcmp at -O3.
ENV CC_x86_64_unknown_linux_gnu=gcc-10 \
CXX_x86_64_unknown_linux_gnu=g++-10 \
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=gcc-10
Comment on lines +22 to +24

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Route x86 CMake builds through GCC 10

In the Linux release path checked in .github/actions/cross-build-binary/action.yml, release-feature-vendored-set builds aws-lc/rdkafka via CMake; the x86_64 cross-rs base image supplies CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu=/opt/toolchain.cmake, and that toolchain chooses ${CROSS_TOOLCHAIN_PREFIX}gcc (x86_64-linux-gnu-gcc) independently of these CC_*/Cargo linker overrides. For the x86_64 GNU artifact, CMake-built deps therefore still use the image's default GCC 9.4 and can hit the aws-lc memcmp compiler rejection unless the CMake compiler or cross toolchain prefix/suffix is switched to gcc-10 too.

Useful? React with 👍 / 👎.


RUN curl -fLO $PBC_URL && \
unzip protoc-21.5-linux-x86_64.zip -d ./protobuf && \
mv ./protobuf/bin/protoc /usr/bin/ && \
Expand Down
Loading