Repository navigation
Conversation
c7093cd to
a1267a7
Compare
|
Sorry for the late review! Could you help rebase it? We have recently upgraded arrow to use 25.0.0. |
|
Is it possible to fix the Windows CI to leverage it? We should also leave at least one CI job to use bundled build to make sure it does not break. |
a1267a7 to
36829ea
Compare
36829ea to
9addcb4
Compare
|
Thanks for the review, @wgtmac. Apologies for the delay getting back to this. The branch has been rebased onto Arrow 25.0.0, and I've incorporated the Hive and C++ linter conversions from #854. Builds that meet the intended outcome (build types are unchanged):
Open exceptions:
More crucially, although this speeds up builds and reduces reliance on warm caches, the extra exceptions above feel like they make CI harder to maintain. I'm leaning toward dropping this change for now because I don't see a simple way to preserve coverage without adding additional complexity to the CI which is already slightly complex. What do you think: should we continue pursuing this change or drop it? |
|
I agree with your judgment. Mixing conda-forge with vcpkg is not a good idea. We should just make it as maintainable as possible. |
| SCCACHE_DIR: ${{ github.workspace }}/.sccache | ||
| SCCACHE_CACHE_SIZE: "2G" | ||
| ICEBERG_EXTRA_CMAKE_ARGS: "-DICEBERG_BUILD_HIVE=ON" | ||
| # Prebuilt Arrow exports no Thrift target, so resolve the Thrift runtime |
There was a problem hiding this comment.
Do we lose the test coverage of hive with bundled thrift?
| mkdir build && cd build | ||
| cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Debug -DICEBERG_ENABLE_ASAN=ON -DICEBERG_ENABLE_UBSAN=ON \ | ||
| -DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache | ||
| if [[ -d _deps/vendoredarrow-src ]]; then |
There was a problem hiding this comment.
can this check be moved into CMake?
|
I share the concern about the maintenance cost: adopting prebuilt Arrow also changes how we obtain Thrift and the AWS SDK, I'd prefer to preserve the existing build configurations and coverage, particularly Windows Debug and Hive with bundled Thrift. The latter currently loses coverage when Hive-enabled jobs switch to system Thrift. Could we first evaluate reusing dependency build artifacts with appropriate cache keys? For Linux, a versioned CI image with prebuilt dependencies could also be worth a small experiment. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Apologies for the delay on this. I've come back to it a few times and gone through your comments, and I think conda is the wrong thing to target. CI will keep changing before 1.0.0, and as @wgtmac mentioned in #799 it should be easy to maintain, and @zhjwpku's comment convinced me this isn't. @manuzhang's question about Hive with bundled Thrift and @wgtmac's ask to keep at least one bundled leg get at the other problem: balancing coverage and CI time gets tricky when we still want some legs running the full from-source build. Going back to the goal in #799, I was wrong that Arrow is most of the cache pressure. It's about 15% of the sccache bytes, debug info is 60 to 80%. Each push to main saves about 10.6 GB against the 10 GB limit, so at least one entry from the previous set gets evicted, and about a quarter of jobs on main since August ran fully cold. Dropping debug info on the Debug legs (still Debug builds, asserts and -O0 stay) got that to about 4.7 GB on my fork with all tests passing, cold builds got faster too, and build types and coverage stay the same. So my leaning is to close this and open a small PR for that instead. What do you all think? |
That sounds perfect to me. Thanks for your continuous investigation on this! @abnobdoss |
|
We can close this PR, as it's unlikely to move forward anyway. @abnobdoss Looking forward to your new fix :-) |
Part of #799 (Cutting CI build times: prebuilt Arrow and build configuration). This is the conda-forge Arrow piece, Unix legs only.
What
Adds
.github/actions/setup-conda-arrow, a composite action that installslibarrowandlibparquetfrom conda-forge and exportsCMAKE_PREFIX_PATHand the loader path.Used on the Unix legs that build Arrow from source today: the
ubuntuandmacosjobs in Test, ASAN/UBSAN, SQL Catalog's non-Windows slices, and the AWS system-SDK slice.Why not the rest
Windows stays vendored because conda-forge only ships Release builds, so a Debug consumer would hit a CRT mismatch (
/MDdvs/MD). That is a packaging limitation rather than something to sequence around.Hive also stays vendored, since prebuilt Arrow exports no
thrift::thrifttarget andiceberg_hiverequires one. I plan to raise that as a follow-up PR.The AWS bundled-SDK slices cannot use conda at all. Bundling makes Arrow build the AWS SDK into its own
arrow_bundled_dependenciesarchive, and that only exists when Arrow is built from source, so only the system-SDK slice is converted. The Meson legs are untouched because they do not use Arrow.Notes
install-arrow.shreadsICEBERG_ARROW_BUILD_VERSIONfromIcebergThirdpartyToolchain.cmakeinstead of repeating the version, so bumping the vendored Arrow moves the conda legs with it. The pin is exact (libarrow==24.0.0), since conda treats=as a prefix match and patch releases do get published.ICEBERG_REQUIRE_SYSTEM_ARROW=ONtellsbuild_iceberg.shto fail the leg if Arrow was built from source anyway. That check is needed becausefetchcontent_declare(VendoredArrow ...)passesFIND_PACKAGE_ARGS: a failedfind_packagefalls back to a source build with no warning, and the leg would still pass.Validation
Converted legs go from 931 to 596 compile requests, and the rest are unchanged.
All 20 checks pass on the fork with the guard armed, so a green run does mean conda Arrow was resolved. Upstream reports the checks as skipped rather than passing, because every job is gated on
draft == false.