Skip to content

Merge upstream apache/iceberg main - #45

Merged
robert3005 merged 98 commits into
mainfrom
rk/merge-upstream-main-2026-11-06
Nov 6, 2026
Merged

robert3005 merged 98 commits into
mainfrom
rk/merge-upstream-main-2026-11-06

Conversation

@robert3005

@robert3005 robert3005 commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Merges apache/iceberg main (up to cd8ea3f) into the fork.

The only conflicts were in the Spark runtime-deps.txt files. These were regenerated with generateRuntimeDeps. Upstream bumped RoaringBitmap to 1.6.25. vortex-jni bundles its own copy of RoaringBitmap, so the class uniqueness baselines are updated.

anoopj and others added 30 commits October 22, 2026 08:36
Plan FileScanTasks from a V4 root manifest, routing DATA entries to tasks and
expanding DATA_MANIFEST entries into their leaf manifests through
V4ManifestReader. Attach each data file's colocated deletion vector as a
DeleteFile. Add RootManifestFile so the root reads through the same reader as
the leaves, and TrackedFileAdapters.asDVDeleteFile(DataFile).
The wrapper still called avgValueSizes() after totals replaced stored averages, so wrapping a data file dropped total_bytes.

Generated-by: Cursor
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
apache#18643)

Bumps [com.github.ben-manes.caffeine:caffeine](https://github.com/ben-manes/caffeine) from 3.3.0 to 3.3.1.
- [Release notes](https://github.com/ben-manes/caffeine/releases)
- [Commits](ben-manes/caffeine@v3.3.0...v3.3.1)

---
updated-dependencies:
- dependency-name: com.github.ben-manes.caffeine:caffeine
  dependency-version: 3.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `jackson-bom` from 2.22.3 to 2.22.4.

Updates `com.fasterxml.jackson.core:jackson-databind` from 2.22.3 to 2.22.4
- [Commits](FasterXML/jackson-databind@jackson-databind-2.22.3...jackson-databind-2.22.4)

Updates `com.fasterxml.jackson:jackson-bom` from 2.22.3 to 2.22.4
- [Commits](FasterXML/jackson-bom@jackson-bom-2.22.3...jackson-bom-2.22.4)

Updates `com.fasterxml.jackson.core:jackson-core` from 2.22.3 to 2.22.4
- [Commits](FasterXML/jackson-core@jackson-core-2.22.3...jackson-core-2.22.4)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-version: 2.22.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-version: 2.22.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.core:jackson-core
  dependency-version: 2.22.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 6.0.1 to 6.1.0.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](actions/setup-java@de7274f...b843ebf)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 6.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
)

Bumps [datamodel-code-generator](https://github.com/datamodel-code-generator/datamodel-code-generator) from 0.82.0 to 0.85.0.
- [Release notes](https://github.com/datamodel-code-generator/datamodel-code-generator/releases)
- [Changelog](https://github.com/datamodel-code-generator/datamodel-code-generator/blob/main/CHANGELOG.md)
- [Commits](datamodel-code-generator/datamodel-code-generator@0.82.0...0.85.0)

---
updated-dependencies:
- dependency-name: datamodel-code-generator
  dependency-version: 0.85.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pymarkdownlnt](https://github.com/jackdewinter/pymarkdown) from 0.9.40 to 0.9.41.
- [Release notes](https://github.com/jackdewinter/pymarkdown/releases)
- [Changelog](https://github.com/jackdewinter/pymarkdown/blob/main/changelog.md)
- [Commits](jackdewinter/pymarkdown@v0.9.40...v0.9.41)

---
updated-dependencies:
- dependency-name: pymarkdownlnt
  dependency-version: 0.9.41
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [com.aliyun:kms20160120](https://github.com/aliyun/alibabacloud-java-sdk) from 2.3.0 to 2.3.1.
- [Changelog](https://github.com/aliyun/alibabacloud-java-sdk/blob/master/docmind-api-20220711/ChangeLog.txt)
- [Commits](https://github.com/aliyun/alibabacloud-java-sdk/commits)

---
updated-dependencies:
- dependency-name: com.aliyun:kms20160120
  dependency-version: 2.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.3.8 to 1.3.9.
- [Release notes](https://github.com/azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-digitaltwins-core_1.3.8...azure-digitaltwins-core_1.3.9)

---
updated-dependencies:
- dependency-name: com.azure:azure-sdk-bom
  dependency-version: 1.3.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…8651)

Bumps [io.grpc:grpc-netty-shaded](https://github.com/grpc/grpc-java) from 1.84.2 to 1.85.0.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.84.2...v1.85.0)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-netty-shaded
  dependency-version: 1.85.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#18650)

Bumps software.amazon.awssdk:bom from 2.55.14 to 2.56.0.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-version: 2.56.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…8394)

* Core: Read geometry and geography bounds as GeospatialBound

Materialize v4 manifest geometry and geography lower and upper bounds as GeospatialBound so callers receive the bound type, and copy reused reader instances when field stats are copied.
Co-authored-by: Claude Code <noreply@anthropic.com>
…v4 (apache#18585)

Track stats tightness with a single file-level tight_stats flag on the
tracked file instead of a per-column tight_bounds field. The flag covers
all statistics (bounds and null, NaN, and value counts) for the file's
live rows.
Use the released maintenance version to include security and correctness fixes.

Generated-by: Codex
Use the released maintenance version to include security and correctness fixes.

Generated-by: Codex
Use the latest Spark 4.2 maintenance release for the integration.

Generated-by: Codex
…pache#18359)

* Core: Copy file-scoped position deletes without stats in DeleteFileIndex

A position delete whose file_path bounds name a single data file is
matched by location, so the copy records the data file in
referencedDataFile and drops column stats.

JOL retained bytes per delete, 100k v2 position deletes, 150-char paths:

| data files | before   | after   |
|------------|----------|---------|
| 100k (1:1) | 2122.6 B | 810.6 B |
| 37k (2.7:1)| 1908.1 B | 717.0 B |

* Core: Share referenced data file locations in DeleteFileIndex

Position deletes of the same data file hold one location String.

JOL retained bytes per delete, 100k v2 position deletes, 150-char paths,
2.7 deletes per data file: 717.0 B before, 596.1 B after.
* Spec: Add table constraints

* nit

* more changes

* fix constraint spec based on the sync

* address comments

* Spec: add constraints and schema evolution section

* clarify constraint status semantics

* clarify constraint validation status semantics

* clarify check expression scope and incremental validation

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* address comments

* move constraint-statuses into v4 snapshot table

Co-authored-by: Cursor <cursoragent@cursor.com>

* address comments

* Remove unvalidated rule for check constraints that call a UDF

* Assume UDFs do not change in check constraints

* remove redudant sentence

* address comments

* address comments

* Remove unknown constraint type rule

* Address Ryan's comments

* Address review comments

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
swapna267 and others added 10 commits November 5, 2026 17:20
…he#18779)

The as-of timestamp was read right after FILE_B's commit, so FILE_C's snapshot could land in the same millisecond and be included. Wait until the clock is past FILE_B's snapshot, and past the timestamp before committing FILE_C.

Generated-by: Claude Code
…umn (apache#18148) (apache#18808)

Filtering a changelog table on _change_type, _change_ordinal or
_commit_snapshot_id failed with a ValidationException because the
predicate was pushed into the Iceberg scan, where these generated
columns do not exist. Leave such predicates for Spark to evaluate after
the scan, as Spark 4.2 already does.
Conflict resolutions:
- Spark runtime-deps.txt: regenerated with generateRuntimeDeps. The
  Vortex dependencies keep Guava and arrow-c-data on the runtime
  classpath.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 40 committers have signed the CLA.

❌ dramaticlly
❌ kevinjqliu
❌ rajarshisarkar
❌ stevenzwu
❌ rdblue
❌ pvary
❌ ebyhr
❌ anoopj
❌ rshdhere
❌ anuragmantri
❌ szehon-ho
❌ zhengxiang378928908-code
❌ huaxingao
❌ raunaqmorarka
❌ manuzhang
❌ huan233usc
❌ Prathameshk2696
❌ jiwen624
❌ randreucetti
❌ venkata91
❌ youngrae81
❌ sjs004
❌ mxm
❌ slfan1989
❌ AlejandroMorgante
❌ swapna267
❌ chenjian2664
❌ kinolaev
❌ adityaksolves
❌ PFischbeck
❌ Guosmilesmile
❌ singhpk234
❌ joyhaldar
❌ thswlsqls
❌ Dzeri96
❌ slachiewicz
❌ yhyyz
❌ kumarpritam863
❌ rahulsmahadev
❌ gaborkaszab
You have signed the CLA already but the status is still pending? Let us recheck it.

@robert3005
robert3005 merged commit 49f2895 into main Nov 6, 2026
42 of 51 checks passed
@robert3005
robert3005 deleted the rk/merge-upstream-main-2026-11-06 branch November 6, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.