Skip to content

ci: purge JAR-less POMs that leave out the packaging element - #6947

Closed
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:ci-purge-default-packaging-poms
Closed

andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:ci-purge-default-packaging-poms

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue. Found while triaging the CI of #6877, whose Spark 3.5 Spark SQL build failed this way.

Rationale for this change

setup-spark-builder restores the Maven repository from Linux-spark-sql-<hash> and falls back to the restore keys Linux-spark-sql-, then Linux-java-maven-. Since #5973, nothing writes spark-sql- entries, so once the last ones are evicted every Spark SQL build starts from the newest java-maven- entry. Since this morning that has been main's Linux-java-maven-<hash>-delta-3.5, saved by the nightly's Delta contrib (Spark 3.5) job. That job downloads the Arrow 12.0.1 POMs for arrow-memory-core and arrow-memory-netty, the Arrow version Spark 3.5 uses, but not their JARs.

The "Purge partial Maven cache entries" step is there to remove such entries before sbt runs. It only removed a JAR-less POM that declares <packaging>jar</packaging> or bundle, and Arrow's POMs have no <packaging> element, so they survived. Coursier took them as found in mavenLocal, and sbt compiled Spark 3.5's sql/api without org.apache.arrow.memory:

[error] .../sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:24:25: object memory is not a member of package org.apache.arrow

A re-run restores the same cache and fails the same way. Today's nightly Spark 3.5 build passed only because a spark-sql- entry still existed when it ran, so the next nightly is exposed too. Spark 4.0 and 4.1 use a newer Arrow, which Comet's own build downloads in full, so their builds pass from the same cache.

What changes are included in this PR?

The purge step now treats a POM with no <packaging> element as a jar, which is Maven's default. POMs that declare any other packaging, such as parent POMs, are still kept.

How are these changes tested?

I ran the step's script, extracted from the action, against a scratch ~/.m2/repository that held the real Arrow 12.0.1 POMs. Before the change it removed neither JAR-less Arrow POM. Now it removes the arrow-memory-core and arrow-memory-netty POMs and still removes a JAR-less POM that declares jar. It keeps the pom-packaging parents (arrow-memory, arrow-java-root) and an entry that has its JAR. actionlint --shellcheck=off and dev/ci/check-ci-config.py pass.

The pull request tier on main does not run the Spark SQL suites. The same commit is on #6877, whose branch-1.1 CI runs the Spark 3.5 Spark SQL build against the same fallback cache.

When no spark-sql- cache exists, the Spark SQL build restores the newest
java-maven- cache instead. That can be the Delta 3.5 job's cache, which
holds the Arrow 12.0.1 POMs for arrow-memory-core and arrow-memory-netty
but not their JARs. The purge step removed a JAR-less POM only if it
declared jar or bundle packaging. Arrow's POMs declare no packaging, so
sbt took them as found in mavenLocal and compiled Spark 3.5's sql/api
without org.apache.arrow.memory.

Treat a POM without a packaging element as a jar, Maven's default.
@github-actions github-actions Bot added build Build environment enhancement New feature or request area:ci CI/CD, GitHub Actions, build tooling labels Oct 13, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff from 55ba50d45137781f50ec1b7323da7109e66f36f9 to 2fad1fadb8fea8c4c03ad4d52cc038bf549e9ef3: one commit changing .github/actions/setup-spark-builder/action.yaml. The PR is not a draft. Existing reviews, conversation comments, inline comments, and threads were empty.

Routed skill: review-comet-pr. No sibling area skill applies to this CI-only change.

No introduced P1/P2 issues found within this review. No substantiated existing blockers remain.

Summary

  • Prior state and problem: The cleanup retained JAR-less POMs that omitted <packaging>. This includes Arrow 12.0.1’s arrow-memory-core and arrow-memory-netty, leaving incomplete local dependencies available to Spark’s build.
  • Design approach: Treat omitted packaging as jar, matching Maven’s default, while retaining the existing checks for explicit packaging and an accompanying JAR.
  • Correctness: Verified Maven’s model declares jar as the default. Spark 3.5.9 source confirms Arrow 12.0.1, the arrow-memory-netty dependency, and the RootAllocator import. Running the extracted cleanup against upstream Arrow POMs reproduced the base behavior and verified the fix. No expression or operator semantics change.
  • Compatibility analysis: Inspected dependency declarations for the configured Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0 versions. The cleanup uses packaging semantics without version-specific branches. Compared the changed file with latest release branch branch-1.1 at 18732aa2150b52e1b456fdb6317fee5947edfb68. Its difference is exactly this intended cache-cleanup fix, with no SQL behavior change.
  • Key design decisions: Entries with JARs bypass cleanup. Explicit non-jar/bundle packaging remains preserved. Cleanup still runs after Comet’s Maven installation and before Spark’s sbt build.
  • Implementation sketch: An outer packaging-presence check makes the existing jar/bundle filter conditional. Eligible POMs and the existing associated metadata files are then removed.
  • Performance: The existing repository traversal and JAR fast path remain unchanged. Explicitly packaged, JAR-less entries incur one additional grep. There is no query-runtime overhead, and no performance defect meeting the reporting bar was identified.
  • Design: The localized condition addresses the demonstrated failure without changing cache keys, build sequencing, or dependency resolution configuration.
  • Abstraction & complexity: No new helper, dependency, or configuration is introduced. The small extension fits the existing cleanup logic. No concrete simplification meeting the reporting bar was identified.
  • Behavioral changes worth calling out: Default-packaging POMs without JARs are now removed so dependencies can be fetched remotely. Parent POMs and complete entries remain available. This changes CI cache preparation, not released query behavior.
  • Suggested improvements: No additional improvement meeting the P1/P2 reporting bar was identified.

Exact-head CI: 7 checks succeeded and 19 were skipped. Successful checks include Preflight, Required Checks, and Actions analysis. Preflight’s workflow lint and CI configuration checks passed. All Spark SQL and Linux/macOS build jobs were skipped, so the green aggregate does not establish an end-to-end build result. CI run.

Validation: dev/ci/check-ci-config.py, YAML parsing, Bash syntax checks, and git diff --check passed locally. Tested 28 disposable cache fixtures against both base and head, covering upstream Arrow POMs, explicit packaging, complete entries, paths with spaces, metadata cleanup, repeated execution, and absent/empty repositories. Only the two repository paths were redirected for isolation. No real Maven cache was modified. Actionlint was unavailable locally but passed in exact-head CI. No full Maven/sbt build or Spark SQL suite was run. The project working tree remains unchanged.

@andygrove andygrove closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI/CD, GitHub Actions, build tooling build Build environment enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants