Repository navigation
Merge upstream apache/iceberg main - #45
Conversation
Plan FileScanTasks from a V4 root manifest, routing DATA entries to tasks and expanding DATA_MANIFEST entries into their leaf manifests through V4ManifestReader. Attach each data file's colocated deletion vector as a DeleteFile. Add RootManifestFile so the root reads through the same reader as the leaves, and TrackedFileAdapters.asDVDeleteFile(DataFile).
The wrapper still called avgValueSizes() after totals replaced stored averages, so wrapping a data file dropped total_bytes. Generated-by: Cursor
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
apache#18643) Bumps [com.github.ben-manes.caffeine:caffeine](https://github.com/ben-manes/caffeine) from 3.3.0 to 3.3.1. - [Release notes](https://github.com/ben-manes/caffeine/releases) - [Commits](ben-manes/caffeine@v3.3.0...v3.3.1) --- updated-dependencies: - dependency-name: com.github.ben-manes.caffeine:caffeine dependency-version: 3.3.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `jackson-bom` from 2.22.3 to 2.22.4. Updates `com.fasterxml.jackson.core:jackson-databind` from 2.22.3 to 2.22.4 - [Commits](FasterXML/jackson-databind@jackson-databind-2.22.3...jackson-databind-2.22.4) Updates `com.fasterxml.jackson:jackson-bom` from 2.22.3 to 2.22.4 - [Commits](FasterXML/jackson-bom@jackson-bom-2.22.3...jackson-bom-2.22.4) Updates `com.fasterxml.jackson.core:jackson-core` from 2.22.3 to 2.22.4 - [Commits](FasterXML/jackson-core@jackson-core-2.22.3...jackson-core-2.22.4) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-version: 2.22.4 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: com.fasterxml.jackson:jackson-bom dependency-version: 2.22.4 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: com.fasterxml.jackson.core:jackson-core dependency-version: 2.22.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 6.0.1 to 6.1.0. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@de7274f...b843ebf) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) Bumps [datamodel-code-generator](https://github.com/datamodel-code-generator/datamodel-code-generator) from 0.82.0 to 0.85.0. - [Release notes](https://github.com/datamodel-code-generator/datamodel-code-generator/releases) - [Changelog](https://github.com/datamodel-code-generator/datamodel-code-generator/blob/main/CHANGELOG.md) - [Commits](datamodel-code-generator/datamodel-code-generator@0.82.0...0.85.0) --- updated-dependencies: - dependency-name: datamodel-code-generator dependency-version: 0.85.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pymarkdownlnt](https://github.com/jackdewinter/pymarkdown) from 0.9.40 to 0.9.41. - [Release notes](https://github.com/jackdewinter/pymarkdown/releases) - [Changelog](https://github.com/jackdewinter/pymarkdown/blob/main/changelog.md) - [Commits](jackdewinter/pymarkdown@v0.9.40...v0.9.41) --- updated-dependencies: - dependency-name: pymarkdownlnt dependency-version: 0.9.41 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [com.aliyun:kms20160120](https://github.com/aliyun/alibabacloud-java-sdk) from 2.3.0 to 2.3.1. - [Changelog](https://github.com/aliyun/alibabacloud-java-sdk/blob/master/docmind-api-20220711/ChangeLog.txt) - [Commits](https://github.com/aliyun/alibabacloud-java-sdk/commits) --- updated-dependencies: - dependency-name: com.aliyun:kms20160120 dependency-version: 2.3.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.3.8 to 1.3.9. - [Release notes](https://github.com/azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-digitaltwins-core_1.3.8...azure-digitaltwins-core_1.3.9) --- updated-dependencies: - dependency-name: com.azure:azure-sdk-bom dependency-version: 1.3.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…8651) Bumps [io.grpc:grpc-netty-shaded](https://github.com/grpc/grpc-java) from 1.84.2 to 1.85.0. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.84.2...v1.85.0) --- updated-dependencies: - dependency-name: io.grpc:grpc-netty-shaded dependency-version: 1.85.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#18650) Bumps software.amazon.awssdk:bom from 2.55.14 to 2.56.0. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.56.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…8394) * Core: Read geometry and geography bounds as GeospatialBound Materialize v4 manifest geometry and geography lower and upper bounds as GeospatialBound so callers receive the bound type, and copy reused reader instances when field stats are copied.
Co-authored-by: Claude Code <noreply@anthropic.com>
…v4 (apache#18585) Track stats tightness with a single file-level tight_stats flag on the tracked file instead of a per-column tight_bounds field. The flag covers all statistics (bounds and null, NaN, and value counts) for the file's live rows.
Use the released maintenance version to include security and correctness fixes. Generated-by: Codex
Use the released maintenance version to include security and correctness fixes. Generated-by: Codex
Use the latest Spark 4.2 maintenance release for the integration. Generated-by: Codex
…pache#18359) * Core: Copy file-scoped position deletes without stats in DeleteFileIndex A position delete whose file_path bounds name a single data file is matched by location, so the copy records the data file in referencedDataFile and drops column stats. JOL retained bytes per delete, 100k v2 position deletes, 150-char paths: | data files | before | after | |------------|----------|---------| | 100k (1:1) | 2122.6 B | 810.6 B | | 37k (2.7:1)| 1908.1 B | 717.0 B | * Core: Share referenced data file locations in DeleteFileIndex Position deletes of the same data file hold one location String. JOL retained bytes per delete, 100k v2 position deletes, 150-char paths, 2.7 deletes per data file: 717.0 B before, 596.1 B after.
* Spec: Add table constraints * nit * more changes * fix constraint spec based on the sync * address comments * Spec: add constraints and schema evolution section * clarify constraint status semantics * clarify constraint validation status semantics * clarify check expression scope and incremental validation * address comments * address comments * address comments * address comments * address comments * address comments * address comments * address comments * address comments * address comments * address comments * address comments * move constraint-statuses into v4 snapshot table Co-authored-by: Cursor <cursoragent@cursor.com> * address comments * Remove unvalidated rule for check constraints that call a UDF * Assume UDFs do not change in check constraints * remove redudant sentence * address comments * address comments * Remove unknown constraint type rule * Address Ryan's comments * Address review comments --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…he#18779) The as-of timestamp was read right after FILE_B's commit, so FILE_C's snapshot could land in the same millisecond and be included. Wait until the clock is past FILE_B's snapshot, and past the timestamp before committing FILE_C. Generated-by: Claude Code
…umn (apache#18148) (apache#18808) Filtering a changelog table on _change_type, _change_ordinal or _commit_snapshot_id failed with a ValidationException because the predicate was pushed into the Iceberg scan, where these generated columns do not exist. Leave such predicates for Spark to evaluate after the scan, as Spark 4.2 already does.
… as a missing table (apache#18582)
…reports it as a missing table (apache#18582) (apache#18814)
Conflict resolutions: - Spark runtime-deps.txt: regenerated with generateRuntimeDeps. The Vortex dependencies keep Guava and arrow-c-data on the runtime classpath.
|
|
Merges
apache/icebergmain(up to cd8ea3f) into the fork.The only conflicts were in the Spark
runtime-deps.txtfiles. These were regenerated withgenerateRuntimeDeps. Upstream bumped RoaringBitmap to 1.6.25.vortex-jnibundles its own copy of RoaringBitmap, so the class uniqueness baselines are updated.