Repository navigation
Remastered - #9
Open
damn-am-star wants to merge 635 commits into
Open
damn-am-star wants to merge 635 commits into
damn-am-star wants to merge 635 commits into
Conversation
## What changes are proposed in this pull request? Previously, `snap.write_checksum` only worked if: - there is a CRC on disk at N, you write `N+1.json`, and then we can write `N+1.crc` - there is a CRC on disk at N - 5, you load a Snapshot with incremental replay turned on, then we have an in-memory CRC at N. Now we can write a CRC at N, or at N+1 if we perform a commit, too Now, this PR enables writing CRC for the cases of: - checkpoint-rooted snapshot [10.checkpoint, 11.json, 12.json] - checkpoint-versioned snapshot [10.checkpoint] - v0 rooted snasphot [0.json, 1.json, 2.json] - stale CRC with incremental replay disabled [10.checkpoint, 11.json, 12.json, 12.crc, 13.json, 14.json] ## How was this change tested? New UTs for these specific cases + error cases. Also added write-checksum to the cross-product suite (2800+ tests) which already covers lots of stale CRC, only checkpoint, no checkpoint, etc cases.
## What changes are proposed in this pull request? Adds a user-guide page documenting the incremental scan API (`Snapshot::incremental_scan_builder`), which advances a cached file listing across a version range without a full log replay. The API exists but had no coverage in the mdBook guide. The page covers the `(base_version, target_version]` range model, the `Option` return and full-scan fallback, `(path, dv_unique_id)` file keying, advancing a cached listing with the `removes` and `duplicate_adds` masks, sticky errors, the vacuum race, the catalog-managed log-tail requirement, and current limitations (no predicate or column pushdown; in-range metadata changes not surfaced, tracked in delta-io#2552). It's added to the "Reading Tables" section of `SUMMARY.md`, after Time Travel and before Change Data Feed. ## How was this change tested? `make test` compiles every Rust example on the page, and `mdbook build` renders it and checks links.
…o#2886) ## What changes are proposed in this pull request? **Issue:** Reading a Delta table fails at snapshot load when the table's column mapping mode is `none` but its schema still carries `delta.columnMapping.physicalName` / `delta.columnMapping.id` field annotations: ``` Schema error: Column mapping is not enabled but field '<x>' is annotated with delta.columnMapping.physicalName ``` Such tables arise when column mapping was enabled and later disabled - the mode reverts to `none` but the residual annotations remain in the schema. While mapping is off these annotations are ignored and the columns resolve by logical name, so the table is well-formed to read, yet kernel rejected it on every read. This diverges from delta-spark behavior. **Fix:** Kernel re-validates column-mapping annotations on every snapshot load, via the shared `make_physical` transform used by both the read and write paths. delta-spark does not: its `createPhysicalSchema` early-returns in `NoMapping` mode, ignoring the annotations. This change aligns the read path with that behavior: 1/ `make_physical` (read path) now tolerates a stale annotation in `None` mode - it resolves the field by its logical name and drops the column-mapping keys from the physical schema, yielding a schema indistinguishable from a table that never carried them. 2/ The write path stays strict: CREATE TABLE and ALTER TABLE still reject stale annotations, so kernel never originates a table in this shape. Because `make_physical` (shared by both paths) is now lenient, the two builders run an explicit strict column-mapping validation pass to preserve that rejection. Making the write path tolerate/normalize such tables to fully match delta-spark is tracked in delta-io#2885. ## How was this change tested? - Unit tests for the validator's ignore-vs-reject behavior in `None` mode (physicalName only, id only, both), and that `make_physical` strips the column-mapping keys while preserving unrelated field metadata, at top level and on a nested leaf. - Integration test that loads a snapshot from a table with residual annotations (mode `none`) and reads data back end to end. - Integration tests that CREATE TABLE and ALTER TABLE still reject stale annotations while mapping is disabled. --------- Co-authored-by: Sanuj Basu <sanujbasu@users.noreply.github.com>
… 1.97 (delta-io#2901) ## What changes are proposed in this pull request? Rust 1.97.0 (2026-07-07) added an `into_array` method, which collides with test-utils' `IntoArray::into_array` and breaks CI. This change rename the trait method to `into_arrow_array`, which resolves the collision. ## How was this change tested? Pure rename, existing tests should suffice.
## What changes are proposed in this pull request? This change adds reader write feature for the adaptive metadata tree as outlined by delta-io/delta#6978 ## How was this change tested?
## What changes are proposed in this pull request? Adds `CheckpointScanShape` (in `kernel/src/checkpoint/`, behind `declarative-plans`), driven by a `PlanExecutor`: resolves a scan's checkpoint topology (`CheckpointShape` -- leaf vs manifest vs none) and, when stats are requested, whether the checkpoint has parsed stats (`CheckpointStatsInfo`). Resolution prefers an applicable `_last_checkpoint` hint (sidecars / checkpoint schema) to avoid reading the checkpoint, and falls back to a footer read or a sidecar-column scan otherwise. Parsed-stats availability is probed once, from the schema of whichever file holds the add actions (the sidecar footer for a manifest, the checkpoint schema for a parquet leaf). Also adds `ParsedLogPath::is_json` to replace ad-hoc extension matching. ## Prior art (delta-io/delta) Spark-Delta doesn't classify checkpoint shape in a single up-front pass; it selects a `CheckpointProvider` subtype and resolves schema lazily per file at scan time. The equivalent logic (pinned to `delta-io/delta@55bf1e8`): - **Topology (leaf / manifest / none)** -- [`Checkpoints.getCheckpointProvider`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/Checkpoints.scala#L96) dispatches by file format; [`CheckpointProvider.apply`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/CheckpointProvider.scala#L104) resolves single-file V1-vs-V2 by `CheckpointMetadata` presence; [`EmptyCheckpointProvider`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/CheckpointProvider.scala#L353) is the no-checkpoint case. A `V2CheckpointProvider` with non-empty sidecars is the manifest case, empty is inline-V2 (leaf-like). - **`_last_checkpoint` hint avoids reads** -- [`LastCheckpointInfo`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/LastCheckpointInfo.scala#L102) (`checkpointSchema`, `v2Checkpoint`) and [`LastCheckpointV2`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/LastCheckpointInfo.scala#L48) (`sidecarFiles`, `nonFileActions`); read by [`readLastCheckpointFile`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/Checkpoints.scala#L369). When the hint embeds sidecars, [`v2ActionsFromLastCheckpointOpt`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/CheckpointProvider.scala#L414) skips the checkpoint read; when it carries `checkpointSchema`, [`getParquetSchema`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/CheckpointProvider.scala#L182) skips the footer read. - **Parsed-stats usability** -- the stats schema comes from the file holding the add actions: the sidecar footer for a manifest (via [`V2CheckpointProvider.allActionsFileIndexesAndSchemas`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/CheckpointProvider.scala#L552), sidecar schema preferring [`CheckpointMetadata.sidecarFileSchema`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/actions/actions.scala#L1639)), or the checkpoint schema for a leaf. The read-time decision is in [`Snapshot.loadActions`](https://github.com/delta-io/delta/blob/55bf1e899785008ca594cae5c4b79386c91c8ec7/spark/src/main/scala/org/apache/spark/sql/delta/Snapshot.scala#L557): use `stats_parsed` iff the add schema has `stats_parsed` and lacks `stats`. Note kernel's `schema_has_compatible_stats_parsed` does a stronger compatibility check against the requested schema, rather than Spark's field-name presence check. ## How was this change tested? New unit tests over real fixtures: V2 json/parquet x manifest/leaf, multi-part leaf, single-part leaf. A `CountingExecutor` asserts the hint fast path performs exactly one sidecar footer read and no checkpoint drain.
## What changes are proposed in this pull request? The `column_name!` macro and friends are very nice for string literals, but string literals are magic constants, subject to all the usual problems: vulnerability to typos, hard to tell what's related vs. merely similar, hard to rename/refactor, etc. Expand the macro to allow 1+ compile-time constant path segments, where: ```rust column_name!(ADD_NAME, "deletionVector.offset") ``` is equivalent to ```rust ColumnName::new([ADD_NAME, "deletionVector", "offset") ``` The generalized rule is that literals continue to split on `.` to become simple path segments (as before), while other constant expressions must be simple path segments (no dots, no special characters). Empty segments and empty paths are rejected -- use `ColumnName::new` if you really want that. ### This PR affects the following public APIs Behavior change: The `column_name!` macro now requires at least one argument and rejects empty path segments. Expanded behavior: The `column_name!` macro now accepts multiple arguments and string constants (not merely string literals). ## How was this change tested? New unit tests and some existing code was ported to use the new macros.
…-io#2661) ## What changes are proposed in this pull request? Some storage systems expose files through an authenticated REST "file API" (list/read/write over HTTP) rather than a plain blob store. This PR adds `RestObjectStore`, an `ObjectStore` over such an API so the default engine can read and write Delta tables through it, plus FFI hooks on the existing engine builder so a connector can construct a REST-backed engine without a per-backend Rust impl. - `AuthHeaderProvider` -- headers attached to every request (auth/identity). Consulted per request so short-lived credentials can be refreshed; `RefreshingHeaderProvider` caches them until shortly before a caller-supplied TTL elapses, re-produce only then. - `RestFileApiContract` -- the REST dialect: URL building, list/put query params, list-response parsing, and HTTP-status-to-error mapping. `RestEndpointConfig` is a data-driven contract configured by path-prefix templates and JSON field names, so many backends need no bespoke Rust. The store implements the operations Delta issues -- get, put, list (with offset), and delete; copy, multipart, and delimited list return `NotSupported`. It re-enforces two guarantees it can't trust the backend for: list entries are verified sorted across pages (a mis-sort errors out rather than silently corrupting log replay), and `list_with_offset`'s exclusive offset is applied client-side. ### FFI engine builder REST wiring lives in `ffi/src/rest_engine.rs` and plugs into the existing `get_engine_builder` / `set_builder_option` / `builder_build` flow: 1. `get_engine_builder(url)` -- for REST, `url` is the service base URL (not a Delta table path). 2. `set_builder_rest_object_store(builder, endpoint_config, callback, context)` -- selects `ObjectStoreBackend::Rest` and passes a `repr(C)` `CRestEndpointConfig` for the file API dialect. 3. `set_builder_option` -- TLS, retries, and static auth via named `REST_BUILDER_OPTION_*` keys (exported to C as `*_KEY` byte arrays). Static `header.<Name>` options apply only when `callback` is null. 4. `builder_build(builder)` -- builds a `RestObjectStore` internally; embedders never call `build_rest_object_store` directly. For dynamic auth, pass a `CAuthHeaderCallback` that fills a `CAuthHeaders` buffer (`headers[0..count]`, `ttl_ms`). When `callback` is null, static headers come from `header.<Name>` builder options. Gated behind the `default-engine` feature; adds optional `async-trait`/`async-stream` deps and a `wiremock` dev-dependency. Compiles against object_store 0.12 (arrow-57) and 0.13 (arrow-58). ### Customization triggers The store stays minimal by default; a consumer opts into resilience and refresh behavior: - `with_max_retries(n)` -- retry transient failures (5xx, connect/timeout) on idempotent requests with backoff. Default `0` (off). - `with_verify_on_ambiguous(true)` -- on an ambiguous `Create` outcome (5xx / dropped connection), read the object back so a write that actually landed is not mistaken for a conflict. Default off. - Auth: `StaticHeaderProvider` (fixed headers) or `RefreshingHeaderProvider` (a closure returning `(headers, Option<ttl>)`; cached until just before the TTL, otherwise re-produce per request). - Contract: `RestEndpointConfig` (data-driven path-prefix templates + JSON field names) or a custom `RestFileApiContract`. Through the FFI builder the same knobs are `set_builder_option` keys: `REST_BUILDER_OPTION_RETRY_MAX_RETRIES`, `REST_BUILDER_OPTION_PUT_VERIFY_ON_AMBIGUOUS`, `REST_BUILDER_OPTION_HEADER_PREFIX` + header name, `REST_BUILDER_OPTION_TLS_*`, plus the auth callback (with optional TTL). ### This PR affects the following public APIs All additive (no breaking changes): `RestObjectStore`, `AuthHeaderProvider`, `StaticHeaderProvider`, `RefreshingHeaderProvider`, `RestFileApiContract`, `RestEndpointConfig`, `RestListPage`, and the REST-related FFI exports (`set_builder_rest_object_store`, `CRestEndpointConfig`, `CAuthHeaders`, `REST_BUILDER_OPTION_*`, etc.). ## How was this change tested? `wiremock`-backed unit tests cover get/put/list/delete, pagination, 404 -> NotFound (empty listing), 409 -> AlreadyExists, cross-page sort enforcement, exclusive-offset filtering, retry, and verified-create. FFI tests exercise `CRestEndpointConfig` conversion, auth callback/TTL collection, builder option keys, and `build_rest_object_store` error paths.
## Stacked PR Splitting delta-io#2812 per review feedback. Each PR is based on `main`; use the incremental link to review only this PR's changes. - **stack/check-constraints-tokenizer** (this PR, delta-io#2882) -- bottom of the stack, so its full diff is already incremental. - stack/check-constraints-parser (delta-io#2883) - stack/check-constraints-lower (delta-io#2884) ## What changes are proposed in this pull request? We add a tokenizer as the first layer for this series of stacked PRs to expand kernel's SQL parser to support predicates. The structure of these PRs is meant for ease of integration when supporting more complex logic in CHECKs down the road ## How was this change tested? new UTs --------- Co-authored-by: Michael Wang <m.wang@databricks.com> Co-authored-by: Isaac <isaac@example.com>
…#2873) ## What changes are proposed in this pull request? Updates the `Testing` guidance in `CLAUDE.md` to suggest `TestTableBuilder` (`test_utils::table_builder`) when a test needs a Delta table in a specific state, rather than hand-rolling commits. - Adds a bullet in the `Testing` section suggesting the builder for feature sets, partitioned/clustered layouts, checkpoints, CRC files, hint states, and post-cleanup logs; loading snapshots via `build_snapshot!`; and `default_sweep` for broad coverage. Notes that lower-level setup like `test_table_setup` (or hand-rolled `add_commit` / `LocalMockTable`) is the fallback for states the builder cannot express, such as corrupt or malformed logs. - Adds `TestTableBuilder` as the leading entry under **Common test helpers -> Table creation in tests**, and clarifies when to reach for the kernel `create_table` builder instead. Docs-only; no code changes. ## How was this change tested? N/A (documentation only).
…a-io#2905) ## What changes are proposed in this pull request? Add an api `snapshot.built_as_latest`, which reflects if the snapshot knows it's latest during build. The api will allow connectors to do log cleanup on latest snapshot only(match spark). <!-- **Uncomment** this section if there are any changes affecting public APIs. Else, **delete** this section. ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? Integration tests covers all combining cases on catalog-managed/non-catalog-managed tables: 1. snapshot is built as latest or not + checkpoint/crc/incremental build/post commit 2. fresh built/incremental built snapshot
## Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2846/files) to review incremental changes. - [**stack/column-defaults-v3-compat**](delta-io#2846) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2846/files)] - [stack/column-defaults-ffi-support](delta-io#2843) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2843/files/9a75abc1762fa8f6a5351fd8ee4ffc8e1afd51ce..e5a973763cf0c8a412d83eb332f6d3d2cae1f2ed)] --------- ## What changes are proposed in this pull request? This PR adds IcebergCompatV3 enforcement of column-default rules. When `icebergCompatV3` is enabled, transaction load now validates every column default and warns on: - non-literal defaults - defaults on non-primitive columns Additionally we move column default validation up to table configuration so if a table has `CURRENT_DEFAULT` metadata that is non-string or the column is variant type but non-null we throw an error. ## How was this change tested? Unit and Integration Testing ## Resources [Tracking Issue](delta-io#2630)
## What changes are proposed in this pull request? Release v0.26.0. Bumps all workspace crates from 0.25.0 to 0.26.0 and regenerates `CHANGELOG.md` / `README.md` via `cargo release`. Crates published from this release (in dependency order): `delta_kernel_derive`, `delta_kernel`, `delta_kernel_default_engine`. See `CHANGELOG.md` for the full list of changes since v0.25.0. ## How was this change tested? Generated by the standard release tooling (`./release.sh 0.26.0`); full workspace test suite passed as part of the pre-commit checks.
## What changes are proposed in this pull request?
Adds optional predicate pushdown (data skipping) to the incremental
scan. A new `IncrementalScanBuilder::with_predicate` restricts the
streamed live `Add` actions to those whose file stats do not provably
exclude the predicate, so a consumer can ask "what files changed in
`(base_version, target_version]` that could match P?" and get back only
the relevant added files instead of every changed file.
Skipping is conservative and reuses the same stats-based filter the read
path uses, so the surviving `Add` set is identical to what a full scan
at the target version with the same predicate would keep among those
added files. Files without stats are always kept, and `Remove` actions
are never filtered (consumers need every removal to evict stale cached
entries). With no predicate, the default path is unchanged.
The change-data-feed path already built a data-skipping filter over raw
`{ add, remove }` action batches by parsing `add.stats` JSON. That
construction is now a shared `DataSkippingFilter::for_raw_action_batch`
helper that both the CDF path and the incremental scan call, so the two
stay in lockstep with no duplicated logic.
## How was this change tested?
Added integration tests in `kernel/tests/incremental_scan.rs`: pushdown
drops non-matching added files, the `None` predicate path is unchanged,
files without stats are kept, removes are never filtered, static
skip-all and tautology predicates behave correctly, and the consumer
against-base classification composes with a predicate. The existing scan
and CDF suites pass unchanged, confirming the shared filter refactor is
behavior-preserving.
## What changes are proposed in this pull request? Adds in calculation methods for computing field IDs for statistics columns in the AMT. AMT stores statistics in individual column and project is by field ID. This provides a helper method to calculate the base statistics ID (statics for each column are grouped together) for a given data or metadata field ID. ## How was this change tested? Added unit tests.
…tes (delta-io#2897) ## What changes are proposed in this pull request? Follow-up to delta-io#2886, which made kernel *read* a column-mapping-`none` table carrying residual `delta.columnMapping.*` field annotations. This closes the write-path half tracked in delta-io#2885 (Gaps 1 and 2): kernel was stricter than delta-spark on writes, so an ALTER on such a table was rejected even when it only touched clean columns, and kernel had no way to normalize these tables. This matches delta-spark's `verifyAndUpdateMetadataChange` rule for `NoMapping` writes. On a `none`-mode CREATE / ALTER: - If the current table has no column-mapping metadata and the write introduces some, the annotations are stripped from the persisted schema (mirroring `dropColumnMappingMetadata`), so kernel never originates a self-inconsistent table. - If the current table already carries residual annotations, they are left in place. This is what lets an ALTER touching only clean columns on a stale table succeed (Gap 1). This replaces the read-side fix's temporary strict-reject on the two write builders with the strip. `Name` / `Id` mode is unchanged. The now-unused `validate_schema_column_mapping_strict` alias is removed. Gap 3 from delta-io#2885 (collapsing the redundant second schema walk by threading the policy through `TableConfiguration::try_new`) is intentionally left for a separate PR. New helpers `schema_has_column_mapping_metadata` and `drop_column_mapping_metadata` in `table_features::column_mapping` mirror delta-spark's `schemaHasColumnMappingMetadata` and `dropColumnMappingMetadata`. Both key off one shared set of column-mapping metadata keys, so detection and stripping stay symmetric. ## How was this change tested? - Unit tests: `schema_has_column_mapping_metadata` detects a field carrying any column-mapping key by presence (id, physicalName, nested ids, parquet field id / nested ids), including physicalName-only and wrong-typed-id cases, and is false for a clean schema; `drop_column_mapping_metadata` removes those keys from every field (top level, nested struct, and array/map leaves) while preserving unrelated metadata. - Integration: CREATE with a stale-annotated schema (`none` mode) now succeeds and persists a stripped schema; ALTER add-column on an already-stale table succeeds and leaves the pre-existing annotation in place while the new field carries none; ALTER add-column that introduces an annotation on a previously-clean table has it stripped. - The read/append tolerance from delta-io#2886 remains covered. --------- Co-authored-by: Sanuj Basu <sanujbasu@users.noreply.github.com>
## What changes are proposed in this pull request? This PR adds fallback capability to PlanBasedEngine. Specifically, it updates the constructor on `PlanBasedEngine` (and associated handlers) such that callers can embed a separate `Engine` impl that will handle any operations which we do not support via plan-based execution. This serves as a compatibility layer such that the existing Engine interfaces all continue to work even when declarative-plans is used for certain APIs. ## How was this change tested? Unit tests.
…delta-io#2906) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2906/files) to review incremental changes. - [**stack/uc-client-models**](delta-io#2906) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2906/files)] ← _this PR_ - [stack/uc-user-agent](delta-io#2909) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2909/files/ea47f4056ad37feff943e939a62f06c2e96b984d..95b2f21c16273531af03afb3478c8a1915a8097a)] - [stack/uc-e2e-smoke](delta-io#2925) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2925/files/95b2f21c16273531af03afb3478c8a1915a8097a..7f22ffb456761b0ff2d956382bb6a2a657b7500c)] - [stack/uc-core](delta-io#2855) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2855/files/7f22ffb456761b0ff2d956382bb6a2a657b7500c..96781da6094eec3bebd7a1dfd86fa30065af3abe)] - [stack/uc-api-v2](delta-io#2826) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2826/files/96781da6094eec3bebd7a1dfd86fa30065af3abe..f981f90fefd45b33cfc59bfbfcadf9c06c4173ff)] --------- ## What changes are proposed in this pull request? This change also adds an `UpdateTableClient` trait, and this change also introduces the wire types for the Unity Catalog Delta-Tables API: 1. `update_table` request models (`Requirement`, `Update`, `UpdateTableRequest`) 2. `load_table` read models (`LoadTableResponse`, `TableMetadata`) These types are added alongside the crate's current models so the crate and its downstream consumers keep compiling; a later PR swaps the read/commit path onto them and removes the existing types. ## How was this change tested? New unit tests, existing tests pass. These wire types were also verified E2E through the UC OSS server at the top of the stack.
…o#2909) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2909/files/ea47f4056ad37feff943e939a62f06c2e96b984d..95b2f21c16273531af03afb3478c8a1915a8097a) to review incremental changes. - [stack/uc-client-models](delta-io#2906) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2906/files)] - [**stack/uc-user-agent**](delta-io#2909) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2909/files/ea47f4056ad37feff943e939a62f06c2e96b984d..95b2f21c16273531af03afb3478c8a1915a8097a)] ← _this PR_ - [stack/uc-e2e-smoke](delta-io#2925) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2925/files/95b2f21c16273531af03afb3478c8a1915a8097a..7f22ffb456761b0ff2d956382bb6a2a657b7500c)] - [stack/uc-core](delta-io#2855) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2855/files/7f22ffb456761b0ff2d956382bb6a2a657b7500c..96781da6094eec3bebd7a1dfd86fa30065af3abe)] - [stack/uc-api-v2](delta-io#2826) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2826/files/96781da6094eec3bebd7a1dfd86fa30065af3abe..f981f90fefd45b33cfc59bfbfcadf9c06c4173ff)] --------- ## What changes are proposed in this pull request? This change adds a `User-Agent` header to requests made by the UC REST client. `ClientConfig` gains a `user_agent` field defaulting to `Delta/<ver> delta-kernel-rs/<ver>`, with a `ClientConfigBuilder::with_user_agent` override. This change replaces the derived `Debug` on `ClientConfig` with a manual impl that redacts the bearer token so it can't leak into logs. ## How was this change tested? New unit tests, existing UC tests pass. This change was also tested E2E on the OSS UC server at the top of the PR stack.
…oad_table (delta-io#2925) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2925/files) to review incremental changes. - [**stack/uc-e2e-smoke**](delta-io#2925) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2925/files)] ← _this PR_ - [stack/uc-log-tail](delta-io#2927) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2927/files/44b31cfc4b08b2452cee7bf63cfa27549949d413..e69233252f7036ff9a0f4174a1794663276a6af0)] - [stack/uc-core](delta-io#2855) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2855/files/e69233252f7036ff9a0f4174a1794663276a6af0..911fa4d66bb4446a8eeb827cd196e61ddaccb4e9)] - [stack/uc-api-v2](delta-io#2826) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2826/files/911fa4d66bb4446a8eeb827cd196e61ddaccb4e9..f3a93309bbd18c365d4a2a7dad06665d4f08be66)] --------- ## What changes are proposed in this pull request? This change adds get_config (protocol handshake) and load_table (catalog-managed table metadata + unpublished commits) to UCClient. This change also adds an E2E test against the OSS UC server, which is now a new CI workflow. ## How was this change tested? New CI tests against OSS UC server, new unit tests for added functions.
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2769/files) to review incremental changes. - [**stack/interval_reads_engine**](delta-io#2769) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2769/files)] ← _this PR_ - [stack/interval-reads-feature](delta-io#2880) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2880/files/21a1fbb3f198f05ba8f40cf29ab57a472832dc91..602480e64fa74a0ef59015d1de282cf4a6fe10a5)] - [stack/interval_write_support](delta-io#2845) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2845/files/602480e64fa74a0ef59015d1de282cf4a6fe10a5..9b410235c9f1ea6820c94b98ab30ccf848759e42)] - [stack/interval_scalars](delta-io#2854) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2854/files/9b410235c9f1ea6820c94b98ab30ccf848759e42..68137785f0369cd0dbf1d42d68cdb433f30aad0b)] - [stack/interval_ffi_changes](delta-io#2903) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2903/files/68137785f0369cd0dbf1d42d68cdb433f30aad0b..5dee84148065e9c5b3513cd6d46b85af9fae28c6)] --------- ## What changes are proposed in this pull request? Support interval types in default engine. Use raw Arrow::Int for interval types, gracefully error everywhere else (Scalars, stats, etc.). We assume raw int32/int64 in Parquet, as per the [RFC](delta-io/delta#7078). This should remain open until RFC gets merged in as an official proposal. ## How was this change tested? Existing `intv_*` tests pass, new unit tests added & existing unit tests modified. --------- Co-authored-by: Ayush Raj <a.raj@databricks.com>
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2880/files/21a1fbb3f198f05ba8f40cf29ab57a472832dc91..602480e64fa74a0ef59015d1de282cf4a6fe10a5) to review incremental changes. - [stack/interval_reads_engine](delta-io#2769) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2769/files)] - [**stack/interval-reads-feature**](delta-io#2880) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2880/files/21a1fbb3f198f05ba8f40cf29ab57a472832dc91..602480e64fa74a0ef59015d1de282cf4a6fe10a5)] ← _this PR_ - [stack/interval_write_support](delta-io#2845) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2845/files/602480e64fa74a0ef59015d1de282cf4a6fe10a5..9b410235c9f1ea6820c94b98ab30ccf848759e42)] - [stack/interval_scalars](delta-io#2854) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2854/files/9b410235c9f1ea6820c94b98ab30ccf848759e42..68137785f0369cd0dbf1d42d68cdb433f30aad0b)] - [stack/interval_ffi_changes](delta-io#2903) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2903/files/68137785f0369cd0dbf1d42d68cdb433f30aad0b..5dee84148065e9c5b3513cd6d46b85af9fae28c6)] --------- ## What changes are proposed in this pull request? Introduce a new reader-writer table feature `intervalType-preview` gated on a feature flag. Enable reads and CDF of Delta tables with this table feature enabled, but keep writes disabled (this will change in the next PR in the stack). ## How was this change tested? New unit tests + integration test. --------- Co-authored-by: Ayush Raj <a.raj@databricks.com>
…elta-io#2926) ## What changes are proposed in this pull request? Expose several internal apis for connectors to do log cleanup <!-- **Uncomment** this section if there are any changes affecting public APIs. Else, **delete** this section. ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? n/a
…elta-io#2927) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2927/files) to review incremental changes. - [**stack/uc-log-tail**](delta-io#2927) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2927/files)] ← _this PR_ - [stack/uc-credentials](delta-io#2934) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2934/files/ca7ca9ec8dc8bcd8d1a5ddbfe513ac53c86bc868..fb90f1d858abde94e30c0206a8ccaa0f2fd3413d)] - [stack/uc-create-wire-types](delta-io#2941) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2941/files/fb90f1d858abde94e30c0206a8ccaa0f2fd3413d..1b95da077071fe133be2d62f94e965299432f2ff)] - [stack/uc-core](delta-io#2855) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2855/files/1b95da077071fe133be2d62f94e965299432f2ff..872dd5191f189d909690657499f502587259732f)] - [stack/uc-api-v2](delta-io#2826) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2826/files/872dd5191f189d909690657499f502587259732f..d671a80f9f394c62f32938bfbffe5598a23517ab)] --------- ## What changes are proposed in this pull request? This change extracts the inline UC commit to LogPath conversion out of `UCKernelClient::load_snapshot_inner` into a standalone function. This refactor helps lay the groundwork for the upcoming Delta-Tables `load_table` API in an upcoming change. ## How was this change tested? New unit tests, existing UC tests pass.
…thod (delta-io#2934) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/2934/files) to review incremental changes. - [**stack/uc-credentials**](delta-io#2934) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2934/files)] ← _this PR_ - [stack/uc-create-wire-types](delta-io#2941) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2941/files/f16e03fe0ff66ec92b98517a1d98d3b12b44ad71..f384c8318061a9212f0ae6c5cf3d9817a9af620b)] - [stack/uc-core](delta-io#2855) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2855/files/f384c8318061a9212f0ae6c5cf3d9817a9af620b..72270d2f0e9e614a661d4797a955c527b5147d8f)] - [stack/uc-api-v2](delta-io#2826) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/2826/files/72270d2f0e9e614a661d4797a955c527b5147d8f..7f3143cb0a1ad4b2368ceabd4729120bf6197a7b)] --------- ## What changes are proposed in this pull request? Adds the Delta Tables API credential surface to the UC client crates. Once the read path swaps with the legacy UC API, we will delete the old credential equivalents. ## How was this change tested? New UT, new integration test against the OSS UC server, existing UC client/kernel tests pass.
…o#2943) ## What changes are proposed in this pull request? In the current implementation of PlanBasedEngine's FFI, it drops the underlying "fallback engine" when the plan-based engine itself is dropped. While this is a valid convention, this complicates the Java side because there are Java-side resources that must also be closed. So then we would have to implement an ownership transfer on Java side to ensure that the handle is not double freed, but the Java resources ARE freed, when the PlanBasedEngine is closed. To simplify this, this PR just changes the ownership semantics such that the consumer must drop both the fallback engine and plan-based engine separately. This simplifies the Java side because the plan-based wrapper can just directly call the fallbaack engine's `close` method (as part of it's own close method). ## How was this change tested? Unit tests
…elta-io#2883) ## Stacked PR Splitting delta-io#2812 per review feedback. Each PR is based on `main`; use the incremental link to review only this PR's changes. - stack/check-constraints-tokenizer (delta-io#2882) - **stack/check-constraints-parser** (this PR, delta-io#2883) -- [incremental changes](https://github.com/delta-io/delta-kernel-rs/pull/2883/files/05391afb9a674f62d032e8ecaff90dc6cef12351..076f5195c7e006821dd2023cd06dcdaa47a134f5) - stack/check-constraints-lower (delta-io#2884) ## What changes are proposed in this pull request? Adds the parser that consumes the token stream from the tokenizer (delta-io#2882) and produces a `Comparison` -- exactly one `operand <op> operand`, including the null-safe `<=>` operator and signed numeric literal (`-234` and '- 234'). Junctions (`AND`/`OR`/`NOT`), parentheses, and `IS [NOT] NULL` are out of grammar and rejected (the caller treats that as "not kernel-parsable"); keyword tokens are rejected wherever they appear. ## How was this change tested? New UTs --------- Co-authored-by: Michael Wang <m.wang@databricks.com> Co-authored-by: Isaac <isaac@example.com> Co-authored-by: Ken Li <kenneth.li87@gmail.com>
## What changes are proposed in this pull request? Exposes the incremental scan API through the C FFI. The kernel API (`Snapshot::incremental_scan_builder`) was reachable only by linking the Rust crate directly; this lets connectors that consume kernel through the C FFI use it too. The surface mirrors the existing `commit_range` and scan iterator conventions: - `snapshot_incremental_scan_builder` builds a range `(base_version, target_version]` from a snapshot. - `incremental_scan_builder_build` returns an `OptionalValue<stream>`, modeling the kernel's `Option` return so callers get the full-scan-fallback signal (`None` when the snapshot's commit list can't cover the range) rather than an error. - `incremental_scan_stream_next_arrow` drains live-Add batches as Arrow C Data Interface data, newest commit first. - `incremental_scan_stream_into_summary` consumes the stream and returns the live-Add and Remove file-key sets, read back via `incremental_scan_summary_visit_live_adds` / `_visit_removes` callbacks that surface each `(path, dv_unique_id)` key. The stream handle wraps the kernel stream in a mutex behind a shared handle (the stream is `Send` but not `Sync`) and retains an engine `Arc` so the reader's runtime outlives the caller's own engine handle. Only `into_summary` is exposed as a terminal for now; the against-base terminals are deferred, and the FFI docs point callers at the manual `removes union (base intersect live_adds)` mask in the meantime. ## How was this change tested? New FFI unit tests in `ffi/src/incremental_scan.rs` cover build to summary, Arrow batch draining, the empty-range error, deletion-vector key round-trip, the double-consume guards, stream abandonment, and drain-on-partial-read. `cargo nextest run -p delta_kernel_ffi --all-features`, clippy, and `cargo doc` pass.
## What changes are proposed in this pull request? Removes the column default in dev feature following delta-io#2846 and enables the feature in kernel. ## How was this change tested? Passing all tests and added some new integration tests. ## Resources [Tracking Issue](delta-io#2630)
## What changes are proposed in this pull request? While building incremental CRC, we had removed the LazyCrc construct. However, that was premature, as we ended up deciding to making incremental CRC replaying during Snapshot construction optional. This PR re-introduces the cached CRC construct. A Snapshot can have a CRC, and that CRC can be at-latest, incrementally-dervived-to-latest, or stale. This PR introduced `SnapshotCrc` which is a wrapper that guards access to the CRC so that callers always know if they are accessing the latest CRC or a potentially-stale one. This is a building block for DomainMetadata and SetTxn queries that are rooted in CRC and in Snapshots that were not built via incremental state replay. ## How was this change tested? New UTs.
## What changes are proposed in this pull request? Treat negative add/remove file sizes as unsafe for incremental CRC file-stat replay. Snapshot loading now degrades file statistics to `Indeterminate` instead of propagating a cast or histogram error. Closes delta-io#2676. ## How was this change tested? - `cargo nextest run -p delta_kernel --lib --all-features negative_size_trips_is_incremental_safe` - `cargo clippy -p delta_kernel --lib --all-features -- -D warnings` - `cargo doc --workspace --all-features --no-deps` The full workspace clippy command currently hits an unrelated `clippy::redundant_closure` error in `ffi/src/ffi_tracing.rs:381` on upstream `main`.
## What changes are proposed in this pull request? The C FFI schema visitor could not construct a kernel schema containing a `void` field because it had no outbound `visit_field_void` entry point. This prevented FFI-based engines from round-tripping an existing VOID-bearing schema, including using that schema as a read projection. This PR adds `visit_field_void`, backed by the existing primitive-field visitor implementation, so engines can add `PrimitiveType::Void` fields through the C boundary. The read-table C example now dispatches VOID fields as well, and its CTest projects `id,void_col` from the existing VOID fixture and compares the generated Arrow output against a checked-in golden file. ## How was this change tested? ```bash cargo +nightly fmt cargo clippy -p delta_kernel_ffi --all-features -- -D warnings cargo nextest run -p delta_kernel_ffi --all-features test_schema_all_types cargo build -p delta_kernel_ffi --features tracing cmake -S ffi/examples/read-table -B ffi/examples/read-table/build cmake --build ffi/examples/read-table/build (cd ffi/examples/read-table/build && ctest -R '^read_and_print_void_col$' --output-on-failure) (cd ffi/examples/read-table/build && ctest --output-on-failure) ```
## What changes are proposed in this pull request? Classifies log-segment failures as `MissingVersion`, `InvalidCheckpoint`, or `InvalidLogSegment` instead of generic errors. Missing-version errors now include the affected version when known, with corresponding FFI mappings and test coverage. ## How was this change tested? All unit tests and integration tests pass.
…tes (delta-io#3249) ## What changes are proposed in this pull request? Note to reviewers: Please don't be intimidated by the 700+ LOC size of this PR. It's about 55% test code, 25% prod code, with the rest comments/blank lines. Incremental snapshot updates normally adopt a newer checkpoint, which removes commit-file metadata covered by that checkpoint from the snapshot. Conflict rebasing instead needs every winning commit since the transaction's read snapshot. For example, a transaction may read snapshot N and attempt to commit at N+1, while commits N+1 through N+5 win first. If a checkpoint exists at N+3, adopting it hides the N+1 and N+2 commit files needed for conflict analysis. This adds `IncrementalSnapshotBuilder::skip_new_checkpoints()`. The opt-in mode still discovers checkpoints, but retains the input snapshot's checkpoint and every contiguous commit after it. `CommitRange::builder_from` can then inspect the complete winning range without listing the log again. Default incremental updates continue to adopt useful checkpoints. Full scans warn when retained history may require extra log replay, and writing a checkpoint clears that state. ### This PR affects the following public APIs - Adds the `IncrementalSnapshotBuilder` alias and its `skip_new_checkpoints()` method. - Adds a source-mode type parameter to `SnapshotBuilder`; `Snapshot::builder_from` now returns `IncrementalSnapshotBuilder`. ## How was this change tested? - Added unit coverage for checkpoint retention, repeated updates, missing commit history, scan warnings, and incremental CRC replay. - Added integration coverage for catalog-provided log tails and snapshot-derived commit ranges that reuse the original listing. - Ran `cargo +nightly fmt --check`, workspace clippy with all features and warnings denied, and workspace documentation checks.
## Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/3268/files) to review incremental changes. - [stack/read-stable-ver](delta-io#3224) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3224/files)] [MERGED] - [stack/builder-ctx](delta-io#3226) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3226/files)] [MERGED] - [stack/rc-metadata-write](delta-io#3229) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3229/files)] [MERGED] - [stack/ack-rc-preserve](delta-io#3252) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3252/files)] [MERGED] - [**stack/emit-preserve-flag**](delta-io#3268) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3268/files)] ← _this PR_ --------- ## What changes are proposed in this pull request? emit row tracking preservation tag in commitInfo ## How was this change tested? integration tests include: - create/alter table txns - row tracking supported, suspended, enabled - connector supply tags in commitInfo or not
## What changes are proposed in this pull request? Publish automatic AI reviews triggered by `pull_request_target` as inline reviews instead of workflow summaries. Explicit `/review` options and workflow-dispatch output modes remain unchanged. Add a source-contract unit test that keeps the automatic default pinned to inline. ## How was this change tested? - Python inline-review unit tests - Workflow YAML parsing - `cargo +nightly fmt` - `cargo clippy --workspace --benches --tests --all-features -- -D warnings` - `cargo doc --workspace --all-features --no-deps`
## Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/3222/files) to review incremental changes. - [**snapshot-hint**](delta-io#3222) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3222/files)] <- _this PR_ - [snapshot-hint-ffi](delta-io#3284) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/3284/files/aa3f34296d2e93a7d257ef633cf53add8dc22b48..2c1a56ec5d7fdccfbc1dfdb6ad46fbd6dc6ae9d9)] --------- ## What changes are proposed in this pull request? This PR adds an `internal-api` path for building a snapshot from complete caller-supplied state. It lets a connector reuse state it already has without asking the engine to list or read the Delta log again. ```rust let hint = SnapshotHint { version, log_segment_files, protocol, metadata, last_checkpoint_hint, crc, version_status: SnapshotHintVersionStatus::Latest, }; let snapshot = Snapshot::builder_for(table_root) .with_snapshot_hint(hint) .build(engine)?; ``` The existing `build(engine)` API is unchanged. The engine is accepted but is not used when a complete hint is present. Kernel still validates the supplied state: - The log segment must be complete and valid for the hinted version. - Protocol and metadata must form a valid table configuration. - A supplied CRC must match the hint version, protocol, and metadata. - Conflicting builder options, such as a log tail or incremental CRC replay, are rejected. - `Latest` marks the snapshot as latest. `Unverified` does not; freshness remains the connector's responsibility. Snapshot metrics now include `load_type = snapshot_hint`. A hinted build emits the normal snapshot success or failure event, but no child log-load or I/O events because it performs no engine I/O. `CountingReporter` exposes this as one counter family labeled by completion status and load type: ```rust let successes = reporter.snapshot_completion_count( SnapshotCompletionStatus::Success, SnapshotLoadType::SnapshotHint, ); ``` The Rust-to-C metric mapping appends the new load type, and the FFI error mapping appends `InvalidSnapshotHint`. ### This PR affects the following public APIs - Under `internal-api`, adds `SnapshotHint`, `SnapshotHintVersionStatus`, and `SnapshotBuilder::with_snapshot_hint`. - Adds `SnapshotLoadType::SnapshotHint` and `Error::SnapshotHint`. - `CountingReporter::snapshot_completion_count` queries the completion counter by `SnapshotCompletionStatus` and `SnapshotLoadType`. - Appends matching FFI enum variants for the new load type and error. ## How was this change tested? ```shell cargo nextest run --locked --workspace --all-features \ -E 'not test(read_table_version_hdfs) and not test(invalid_handle_code)' cargo +nightly fmt cargo clippy --workspace --benches --tests --all-features -- -D warnings cargo doc --workspace --all-features --no-deps ```
) ## What changes are proposed in this pull request? Kernel currently has a lot of infrastructure to provide the `EvaluationHandler::create_one` convenience wrapper. It predates both `EvaluationHandler::create_many` and more recently-added struct-to-scalar conversion capability. Remove all that legacy cruft and use `create_many` everywhere. Introduce a new freestanding (one-liner) `create_row` function in its place. Also remove `EvaluationHandler::create_null`, which has only one prod use site. While we're at it, change `create_many` signature to require owned input, observing that all callers could easily provide owned data (all own the input and none need to reuse it afterward; some have to manually extract borrowed slices from their owned values in order to satisfy the current API). ### This PR affects the following public APIs `EvaluationHandler::create_many` now requires owned `Vec<Vec<Scalar>>`. Removed `EvaluationHandler::create_null` method. Use `create_many` instead. The `EvaluationHandlerExtension` trait was removed. Use the new `create_row` helper instead. The `IntoEngineData` trait was removed, along with its companion derive macro. Derive `IntoStructData` instead. The `Error::LiteralExpressionTransformError` variant was removed and FFI error updated to match. ## How was this change tested? Lots of unit tests cover this code.
## What changes are proposed in this pull request? Small change. Validate that multipart checkpoints declare at least two parts and contain unique part numbers within the declared range. Fixes delta-io#3297. Follow-up to: delta-io#3222 (comment) ## How was this change tested? - Focused multipart checkpoint tests - Workspace clippy and docs - Full workspace test suite
## What changes are proposed in this pull request? The MODIFIED status is not part of the RFC: https://github.com/delta-io/delta/blob/master/protocol_rfcs/iceberg-v4-metadata.md#tracking <!-- **Uncomment** this section if there are any changes affecting public APIs. Else, **delete** this section. ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested?
To align the code with the docs: https://github.com/delta-io/delta/blob/master/protocol_rfcs/iceberg-v4-metadata.md#tracking ## What changes are proposed in this pull request? <!-- **Uncomment** this section if there are any changes affecting public APIs. Else, **delete** this section. ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested?
## What changes are proposed in this pull request? This updates the experimental AI reviewer in four ways: - Treat source TODOs as durable review state. A defect already described by a nearby TODO or FIXME with a concrete issue reference, such as `TODO(delta-io#3297)`, is omitted. New or changed untracked TODOs are reported as non-blocking notes unless the incomplete behavior is blocking. - Read bounded prior AI review history before each run. The orchestrator and sub-agents omit semantically duplicate findings, while publication independently suppresses exact duplicate comments. History is accepted only from the expected bot login with the AI review marker, sorted newest-first across comments and reviews, and treated as untrusted text. The GitHub Actions bot is trusted by default; a named Omnigent App can be added with `OMNIGENT_BOT_LOGIN`. - Avoid duplicating inline findings in the main review body. Findings attached to diff lines and exact duplicates of prior inline comments are removed from the collapsed body; findings that cannot be mapped remain there with the overall summary. A valid mapped machine finding is still published if its prose heading is missing. Malformed Markdown fails open so body cleanup cannot delete unrelated findings or summary text. - Publish the final complete revision from a multi-turn reviewer transcript. Marker pairs must be complete and non-nested; malformed sequences still fail closed. The orchestrator also tracks every dispatched reviewer before emitting a marked result. Finding IDs are run-local and are not used for cross-run matching. If GitHub history retrieval fails, review continues without cross-run deduplication. The end-to-end behavior is being exercised on draft PR delta-io#3300. That PR contains intentional review targets and must not be merged. Prompt-source deduplication is intentionally separate in draft PR delta-io#3305. ## How was this change tested? - 55 AI reviewer Python tests, including spoofed bot identities, malformed and non-UTF-8 history, chronological history limits, side-aware exact duplicates, off-diff findings, malformed Markdown, balanced-fence recovery, corrected review revisions, reviewer-output grammar, and inline/body separation - Python compilation and YAML parsing for the workflow and reviewer configs - Live GraphQL validation of the GitHub Actions bot identity - Exact-branch workflow runs against draft PR delta-io#3300: tracked-TODO suppression, semantic and exact duplicate suppression, inline findings omitted from the collapsed body, and neutral check publication - Final exact-branch inline run at `10d6ca35`: https://github.com/delta-io/delta-kernel-rs/actions/runs/34559393303 - Full repository pre-commit checks: nightly rustfmt, Clippy, Rust tests and doctests, and coverage
## What changes are proposed in this pull request? Promote an element-level cast discovered while planning a Parquet list read into a cast of the complete list column. Arrow then recursively converts the values while preserving the list wrapper, instead of attempting the invalid `List<Int32> -> Int64` cast. The regression test recreates the reported table shape in memory: a V0 Parquet file stores `array<int>`, V1 changes the table schema to `array<long>`, and the scan verifies the older values are returned as `Int64`. Closes delta-io#2188. ## How was this change tested? - `cargo nextest run -p delta_kernel --test integration --all-features type_widening` - `cargo clippy -p delta_kernel --all-targets --all-features -- -D warnings` - `cargo doc --workspace --all-features --no-deps`
## What changes are proposed in this pull request? Use each reviewer bundle's `REVIEW.md` as its executable Omnigent prompt through `instructions: REVIEW.md`. This removes the duplicate prompt body from every `config.yaml` while keeping the Markdown files directly usable by local review agents. The Markdown bodies match the hardened inline prompts from the rebased parent, preserving bounded read-only source-tool access, offline Delta protocol references, and untrusted-source guidance. Runtime validation loads every reviewer bundle and verifies that its resolved instructions match the adjacent Markdown file. Operator guidance now lives in `reviewer/README.md`. This is a cleanup follow-up to delta-io#3299 and remains separate from that behavior change. ## How was this change tested? - 63 AI reviewer Python tests - YAML parsing for every reviewer config and the workflow - Exact comparison of all 8 Markdown prompts with the rebased parent prompts - Exact-branch artifact run against draft experiment PR delta-io#3300: https://github.com/delta-io/delta-kernel-rs/actions/runs/34648053845
## What changes are proposed in this pull request? This PR exposes typed snapshot hints through the C FFI. Connectors can provide the complete state needed to construct a snapshot - including log paths, protocol, metadata, checkpoint state, and optional CRC state - using `snapshot_builder_set_snapshot_hint`. The setter copies borrowed FFI inputs into owned kernel values. During build, kernel validates the log-segment shape, table configuration, and CRC consistency, then constructs the snapshot without storage I/O. The connector remains responsible for path provenance and its latest-version claim. ### This PR affects the following public APIs - Adds `FfiSnapshotHint` and its nested borrowed FFI types. - Adds `snapshot_builder_set_snapshot_hint`. - Adds snapshot-hint reconstruction APIs under the experimental `internal-api` feature. Follow-up validation and FFI consistency work is tracked in delta-io#3309, delta-io#3310, and delta-io#3311. ## How was this change tested? ```shell cargo nextest run -p delta_kernel_ffi --all-features cargo nextest run -p delta_kernel --lib --all-features crc cargo nextest run -p delta_kernel --lib --all-features snapshot_hint cargo +nightly fmt --check cargo clippy --workspace --benches --tests --all-features -- -D warnings cargo doc --workspace --all-features --no-deps ```
…ta-io#3313) ## What changes are proposed in this pull request? doc the type widening expectation for `ParquetHandler` impl resolves delta-io#1612 <!-- **Uncomment** this section if there are any changes affecting public APIs. Else, **delete** this section. ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? doc only
## What changes are proposed in this pull request? This PR adds a new tracing `Layer` which emits "frame" events. Within kernel functions, we can opt-in a callsite by adding an `enable_call_frame` field on the tracing annotation. When the layer receives span events with such a field, it will emit an OPEN/CLOSE signal so that subscribers can build up a frame profile using timings in between events. The expectation is that the reporting callback runs on the same thread as the invoked function, so that subscribers can maintain a frame stack in some sort of thread-local context. This PR also adds the necessary FFI bindings, though it does not explicitly enable any call sites to emit call frames. ## How was this change tested? Unit tests
## What changes are proposed in this pull request? Harden the Kernel release workflow: - allow maintainers to select an alternate Cargo registry without editing `release.sh` - add an idempotent changelog refresh command plus CI verification so PRs merged while a release PR is open cannot be silently omitted - restrict Kernel release boundaries to semantic-version tags so artifact tags such as `v0.0.1_dat` cannot truncate the changelog Changelog verification uses git-cliff's own filtered commit context and reachable branch tags, runs only on `release/` branches, and fails closed if its tooling cannot resolve the release range. A failed refresh restores the existing changelog. This complements delta-io#3066, which handles independently versioned UC crates. Its suffixed per-crate tags are deliberately excluded from Kernel changelog boundaries. ## How was this change tested? - `.github/scripts/test-release.sh` with git-cliff 2.10.1, covering alternate-registry forwarding, artifact and divergent tags, late PRs, skipped release commits, idempotent refresh, and failure restoration - `./release.sh verify-changelog` - `bash -n` and ShellCheck 0.10.0 - actionlint 1.7.7 - `cargo +nightly fmt -- --check` - `cargo doc --workspace --all-features --no-deps`
…o#3274) ## What changes are proposed in this pull request? Short term solution mentioned in delta-io#3263 <!-- **Uncomment** this section if there are any changes affecting public APIs. Else, **delete** this section. ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? positive test: all kernel data types; nested mismatches negative test: all validation failure cases including missing, mismatching fields
) ## What changes are proposed in this pull request? Followup to delta-io#3306. Adds tracing annotations in order for kernel-rs to start emitting frame graph events on notable call sites. Specifically targeting 1) Scan and 2) Snapshot construction. ## How was this change tested? Tested against e2e prototype
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
How was this change tested?