Skip to content

Decide and document the delta-spark version pin for the Delta contrib #5390

Open

Description

@schenksj

Follow-up from review of #4952 (tracking comment, item 3).

What is pinned today

The contrib-delta Maven profile pins a delta-spark version per Spark profile, in spark/pom.xml:

Spark profile delta.version
spark-3.5 3.3.2
spark-4.0 4.0.0
spark-4.1 4.1.0

with a top-level default of 4.1.0 in the root pom.xml. dev/verify-contrib-delta-gate.sh
asserts the mapping holds (spark-4.1 -> delta-spark 4.1.x, spark-3.5 -> 3.x, spark-4.0 -> 4.0.x)
and fails the build gate otherwise.

The gap

The pin is a real compatibility decision — the contrib reflects into delta-spark internals
(TahoeLogFileIndex, PreparedDeltaFileIndex, TahoeBatchFileIndex, DV plumbing), so the
supported range is narrower than "any delta-spark". But that decision is currently only implicit
in the poms plus an assertion in the gate script. There is no issue recording:

  • which delta-spark versions the contrib claims to support, and why those;
  • what a user should expect if they run against a different delta-spark than the pinned one;
  • who updates the pin when a new delta-spark ships, and what has to be re-verified (the
    reflection surface is the risky part, not the wire format);
  • whether the per-Spark mapping should ever become a range rather than an exact pin.

Ask

Settle the policy and write it down — in the contrib user guide and/or the profile comments — and
keep the gate script assertion as the enforcement. This is a documentation/decision issue rather
than a code change; the current pins work.


🤖 Filed with Claude Code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciCI/CD, GitHub Actions, build toolingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions