You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Decide and document the delta-spark version pin for the Delta contrib #5390
The contrib-delta Maven profile pins a delta-spark version per Spark profile, in spark/pom.xml:
Spark profile
delta.version
spark-3.5
3.3.2
spark-4.0
4.0.0
spark-4.1
4.1.0
with a top-level default of 4.1.0 in the root pom.xml. dev/verify-contrib-delta-gate.sh
asserts the mapping holds (spark-4.1 -> delta-spark 4.1.x, spark-3.5 -> 3.x, spark-4.0 -> 4.0.x)
and fails the build gate otherwise.
The gap
The pin is a real compatibility decision — the contrib reflects into delta-spark internals
(TahoeLogFileIndex, PreparedDeltaFileIndex, TahoeBatchFileIndex, DV plumbing), so the
supported range is narrower than "any delta-spark". But that decision is currently only implicit
in the poms plus an assertion in the gate script. There is no issue recording:
which delta-spark versions the contrib claims to support, and why those;
what a user should expect if they run against a different delta-spark than the pinned one;
who updates the pin when a new delta-spark ships, and what has to be re-verified (the
reflection surface is the risky part, not the wire format);
whether the per-Spark mapping should ever become a range rather than an exact pin.
Ask
Settle the policy and write it down — in the contrib user guide and/or the profile comments — and
keep the gate script assertion as the enforcement. This is a documentation/decision issue rather
than a code change; the current pins work.
Activity